Scott Sullivan
2018-Oct-02 12:42 UTC
[libvirt-users] Libvirt 4.2.0 hang on destination on live migration cancel
I'm trying the following command on the source machine:
virsh migrate --live --copy-storage-all --verbose TEST qemu+ssh://
10.30.76.66/system
If I ssh into the destination machine when this command is running, I can
see NBD copying data as expected, and if I wait long enough it completes
and succeeds.
However if I ctrl+c this command above before it completes, it causes virsh
commands (like virsh list) to just hang seemingly forever on the
destination machine.
Here's the libvirt debug entries when I run virsh list in this state:
[root@host ~]# export LIBVIRT_DEBUG=1
[root@host ~]# virsh list
2018-10-02 05:14:05.053+0000: 16207: debug : virGlobalInit:359 : register
drivers
2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:657 :
driver=0x7f70c1ddbb60 name=Test
2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 :
registering Test as driver 0
2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:657 :
driver=0x7f70c1ddc660 name=ESX
2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 :
registering ESX as driver 12018-10-02 05:14:05.054+0000: 16207: debug :
virRegisterConnectDriver:657 : driver=0x7f70c1ddd560 name=remote
2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 :
registering remote as driver 2
2018-10-02 05:14:05.054+0000: 16207: debug :
virEventRegisterDefaultImpl:280 : registering default event implementation
2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollAddHandle:115 :
Used 0 handle slots, adding at least 10 more
2018-10-02 05:14:05.054+0000: 16207: debug :
virEventPollInterruptLocked:722 : Skip interrupt, 0 0
2018-10-02 05:14:05.054+0000: 16207: info : virEventPollAddHandle:140 :
EVENT_POLL_ADD_HANDLE: watch=1 fd=4 events=1 cb=0x7f70c18791c0 opaque=(nil)
ff=(nil)
2018-10-02 05:14:05.054+0000: 16207: debug : virEventRegisterImpl:241 :
addHandle=0x7f70c187a8a0 updateHandle=0x7f70c1879500
removeHandle=0x7f70c1879360 addTimeout=0x7f70c187a660
updateTimeout=0x7f70c1879690 removeTimeout=0x7f70c1879220
2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollAddTimeout:230 :
Used 0 timeout slots, adding at least 10 more2018-10-02 05:14:05.054+0000:
16207: debug : virEventPollInterruptLocked:722 : Skip interrupt, 0 0
2018-10-02 05:14:05.054+0000: 16207: info : virEventPollAddTimeout:253 :
EVENT_POLL_ADD_TIMEOUT: timer=1 frequency=-1 cb=0x5559f8dba380
opaque=0x7ffe5b15e700 ff=(nil)
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenAuth:1218 :
name=<null>, auth=0x7f70c1ddea00, flags=0x0
2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW:
obj=0x5559fa83a3f0 classname=virConnect
2018-10-02 05:14:05.054+0000: 16208: debug :
virThreadJobSet:99 : Thread 16208 is now running job vshEventLoop
2018-10-02 05:14:05.054+0000: 16208: debug : virEventRunDefaultImpl:324 :
running default event implementation
2018-10-02 05:14:05.054+0000: 16207: debug : virConfLoadConfig:1576 :
Loading config file '/etc/libvirt/libvirt.conf'
2018-10-02 05:14:05.054+0000: 16208: debug :
virEventPollCleanupTimeouts:525 : Cleanup 1
2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollCleanupHandles:574
: Cleanup 1
2018-10-02 05:14:05.054+0000: 16207: debug : virConfReadFile:752 :
filename=/etc/libvirt/libvirt.conf
2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollMakePollFDs:401 :
Prepare n=0 w=1, f=4 e=1 d=0
2018-10-02 05:14:05.054+0000: 16208: debug :
virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers
2018-10-02 05:14:05.054+0000: 16208: debug :
virEventPollCalculateTimeout:371 : No timeout is pending
2018-10-02 05:14:05.054+0000: 16208: info : virEventPollRunOnce:640 :
EVENT_POLL_RUN: nhandles=1 timeout=-1
2018-10-02 05:14:05.054+0000: 16207: debug : virFileClose:110 : Closed fd 6
2018-10-02 05:14:05.054+0000: 16207: debug : virConfGetValueString:897 :
Get value string (nil) 0
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1013 :
no name, allowing driver auto-select
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 :
trying driver 0 (Test) ...
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1071 :
driver 0 Test returned DECLINED
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 :
trying driver 1 (ESX) ...
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1071 :
driver 1 ESX returned DECLINED
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 :
trying driver 2 (remote) ...
2018-10-02 05:14:05.054+0000: 16207: debug : remoteConnectOpen:1350 :
Auto-probe remote URI
2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:916 : proceeding
with name 2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:925 :
Connecting
with transport 1
2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:1060 : Proceeding
with sockname /var/run/libvirt/libvirt-sock
2018-10-02 05:14:05.054+0000: 16207: debug : virNetSocketNewConnectUNIX:641
: path=/var/run/libvirt/libvirt-sock spawnDaemon=0 binary=<null>
2018-10-02 05:14:05.054+0000: 16207: debug : virNetSocketNewConnectUNIX:705
: connect() succeeded
2018-10-02 05:14:05.054+0000: 16207: debug : virNetSocketNew:237 :
localAddr=0x7ffe5b15dd80 remoteAddr=0x7ffe5b15dcf0 fd=6 errfd=-1 pid=0
2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW:
obj=0x5559fa83aa20 classname=virNetSocket
2018-10-02 05:14:05.054+0000: 16207: info : virNetSocketNew:293 :
RPC_SOCKET_NEW: sock=0x5559fa83aa20 fd=6 errfd=-1 pid=0
localAddr=127.0.0.1;0, remoteAddr=127.0.0.1;0
2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW:
obj=0x5559fa83ae20 classname=virNetClient
2018-10-02 05:14:05.055+0000: 16207: info : virNetClientNew:328 :
RPC_CLIENT_NEW: client=0x5559fa83ae20 sock=0x5559fa83aa20
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF:
obj=0x5559fa83ae20
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF:
obj=0x5559fa83aa20
2018-10-02 05:14:05.055+0000: 16207: debug :
virEventPollInterruptLocked:726 : Interrupting
2018-10-02 05:14:05.055+0000: 16207: info : virEventPollAddHandle:140 :
EVENT_POLL_ADD_HANDLE: watch=2 fd=6 events=1 cb=0x7f70c19b4df0
opaque=0x5559fa83aa20 ff=0x7f70c19b5320
2018-10-02 05:14:05.055+0000: 16207: debug : virKeepAliveNew:196 :
client=0x5559fa83ae20, interval=-1, count=0
2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW:
obj=0x5559fa83b120 classname=virKeepAlive
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollRunOnce:650 : Poll
got 1 event(s)
2018-10-02 05:14:05.055+0000: 16207: info : virKeepAliveNew:215 :
RPC_KEEPALIVE_NEW: ka=0x5559fa83b120 client=0x5559fa83ae20
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollDispatchTimeouts:432 : Dispatch 1
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF:
obj=0x5559fa83ae20
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollDispatchHandles:478 : Dispatch 1
2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW:
obj=0x5559fa83b340 classname=virConnectCloseCallbackData
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollDispatchHandles:492 : i=0 w=1
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF:
obj=0x5559fa83b340
2018-10-02 05:14:05.055+0000: 16208: info : virEventPollDispatchHandles:506
: EVENT_POLL_DISPATCH_HANDLE: watch=1 events=1
2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW:
obj=0x5559fa83b040 classname=virNetClientProgram
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollCleanupTimeouts:525 : Cleanup 1
2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW:
obj=0x5559fa83ac00 classname=virNetClientProgram
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574
: Cleanup 2
2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW:
obj=0x5559fa83a9b0 classname=virNetClientProgram
2018-10-02 05:14:05.055+0000: 16208: debug : virEventRunDefaultImpl:324 :
running default event implementation
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF:
obj=0x5559fa83b040
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollCleanupTimeouts:525 : Cleanup 1
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF:
obj=0x5559fa83ac00
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574
: Cleanup 2
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF:
obj=0x5559fa83a9b0
2018-10-02 05:14:05.055+0000: 16207: debug : doRemoteOpen:1179 : Trying
authentication
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 :
Prepare n=0 w=1, f=4 e=1 d=0
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 :
Prepare n=1 w=2, f=6 e=1 d=0
2018-10-02 05:14:05.055+0000: 16207: debug : virNetMessageNew:46 :
msg=0x5559fa83b720 tracked=0
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollCalculateTimeout:371 : No timeout is pending
2018-10-02 05:14:05.055+0000: 16207: debug : virNetMessageEncodePayload:389
: Encode length as 28
2018-10-02 05:14:05.055+0000: 16208: info : virEventPollRunOnce:640 :
EVENT_POLL_RUN: nhandles=2 timeout=-1
2018-10-02 05:14:05.055+0000: 16207: info : virNetClientSendInternal:2121 :
RPC_CLIENT_MSG_TX_QUEUE: client=0x5559fa83ae20 len=28 prog=536903814 vers=1
proc=66 type=0 status=0 serial=0
2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientCallNew:2074 : New
call 0x5559fa83b6a0: msg=0x5559fa83b720, expectReply=1, nonBlock=0
2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientIO:1880 : Outgoing
message prog=536903814 version=1 serial=0 proc=66 type=0 length=28
dispatch=(nil)
2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientIO:1939 : We have
the buck head=0x5559fa83b6a0 call=0x5559fa83b6a0
2018-10-02 05:14:05.055+0000: 16207: info : virEventPollUpdateHandle:152 :
EVENT_POLL_UPDATE_HANDLE: watch=2 events=0
2018-10-02 05:14:05.055+0000: 16207: debug :
virEventPollInterruptLocked:726 : Interrupting
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollRunOnce:650 : Poll
got 1 event(s)
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollDispatchTimeouts:432 : Dispatch 1
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollDispatchHandles:478 : Dispatch 2
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollDispatchHandles:492 : i=0 w=1
2018-10-02 05:14:05.055+0000: 16208: info : virEventPollDispatchHandles:506
: EVENT_POLL_DISPATCH_HANDLE: watch=1 events=1
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollCleanupTimeouts:525 : Cleanup 1
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574
: Cleanup 2
2018-10-02 05:14:05.055+0000: 16208: debug : virEventRunDefaultImpl:324 :
running default event implementation
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollCleanupTimeouts:525 : Cleanup 1
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574
: Cleanup 2
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 :
Prepare n=0 w=1, f=4 e=1 d=0
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 :
Prepare n=1 w=2, f=6 e=0 d=0
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollCalculateTimeout:371 : No timeout is pending
2018-10-02 05:14:05.055+0000: 16208: info : virEventPollRunOnce:640 :
EVENT_POLL_RUN: nhandles=1 timeout=-1
And here's some information from gbd when its in this state:
[root@host ~]# ps axf | grep libvirtd
17010 pts/2 S+ 0:00 \_ grep libvirtd
15555 ? Sl 0:00 libvirtd --daemon
[root@host ~]# gdb -p $(pgrep libvirtd)
(gdb) t a a bt
Thread 17 (Thread 0x2ae648a10700 (LWP 15559)):
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>,
m=<value
optimized out>) at util/virthread.c:154
#2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x2ae648c11700 (LWP 15560)):
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>,
m=<value
optimized out>) at util/virthread.c:154
#2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
---Type <return> to continue, or q <return> to quit---
#4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x2ae648e12700 (LWP 15561)):
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>,
m=<value
optimized out>) at util/virthread.c:154
#2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x2ae649013700 (LWP 15562)):
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>,
m=<value
optimized out>) at util/virthread.c:154
---Type <return> to continue, or q <return> to quit---
#2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x2ae649214700 (LWP 15563)):
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>,
m=<value
optimized out>) at util/virthread.c:154
#2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x2ae649415700 (LWP 15564)):
---Type <return> to continue, or q <return> to quit---
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>,
m=<value
optimized out>) at util/virthread.c:154
#2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x2ae649616700 (LWP 15565)):
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>,
m=<value
optimized out>) at util/virthread.c:154
#2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
Thread 10 (Thread 0x2ae649817700 (LWP 15566)):
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>,
m=<value
optimized out>) at util/virthread.c:154
#2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x2ae649a18700 (LWP 15567)):
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>,
m=<value
optimized out>) at util/virthread.c:154
#2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
---Type <return> to continue, or q <return> to quit---
#4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x2ae649c19700 (LWP 15568)):
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>,
m=<value
optimized out>) at util/virthread.c:154
#2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x2ae64d0aa700 (LWP 15588)):
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>,
m=<value
optimized out>) at util/virthread.c:154
---Type <return> to continue, or q <return> to quit---
#2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x2ae64d2ab700 (LWP 15589)):
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>,
m=<value
optimized out>) at util/virthread.c:154
#2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x2ae64d4ac700 (LWP 15590)):
---Type <return> to continue, or q <return> to quit---
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>,
m=<value
optimized out>) at util/virthread.c:154
#2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x2ae64d6ad700 (LWP 15591)):
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>,
m=<value
optimized out>) at util/virthread.c:154
#2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
Thread 3 (Thread 0x2ae64d8ae700 (LWP 15592)):
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>,
m=<value
optimized out>) at util/virthread.c:154
#2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x2ae64dec8700 (LWP 15632)):
#0 0x00002ae63d97ed0d in recvmsg () from /lib64/libpthread.so.0
#1 0x00002ae63e9caef0 in udev_monitor_receive_device () from
/lib64/libudev.so.0
#2 0x00002ae64be26864 in udevEventHandleThread (opaque=<value optimized
out>) at node_device/node_device_udev.c:1623
#3 0x00002ae63b2c0075 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
---Type <return> to continue, or q <return> to quit---
#4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2ae6429759e0 (LWP 15555)):
#0 0x00002ae63d97e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00002ae63d9795d8 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00002ae63d9794a7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00002ae64be2364b in udevEventHandleCallback (watch=<value optimized
out>, fd=19, events=<value optimized out>, data=<value optimized
out>) at
node_device/node_device_udev.c:1666
#4 0x00002ae63b257564 in virEventPollDispatchHandles () at
util/vireventpoll.c:508
#5 virEventPollRunOnce () at util/vireventpoll.c:657
#6 0x00002ae63b255eb0 in virEventRunDefaultImpl () at util/virevent.c:327
#7 0x00002ae63b3a338d in virNetDaemonRun (dmn=0x55b1d39818d0) at
rpc/virnetdaemon.c:858
#8 0x000055b1d2a4d60b in main (argc=<value optimized out>, argv=<value
optimized out>) at remote/remote_daemon.c:1475
(gdb)
This issue I know is fixed in 4.7.0 because I run this version on my CentOS
7 machines and don't have this issue. However on my CentOS 6 machines I
have 4.2.0, and do experience this issue. Upgrading to 4.7.0 on CentOS 6 is
troublesome though, because of the requirement of gnutls >= 3.2.
I'm hoping with the above output someone might be able to point me in the
right direction to a commit that resolved this between 4.2.0 and 4.7.0, or
a possible way to prevent this hang in 4.2.0 as described above ?
Thanks in advance
Scott Sullivan
2018-Oct-03 02:02 UTC
Re: [libvirt-users] Libvirt 4.2.0 hang on destination on live migration cancel
For anyone who might find this thread from online searching, I was able to determine this commit fixes the above issue: https://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=dddcb601ebf97ef222a03bb27b2357e831e8a0cc On Tue, Oct 2, 2018 at 8:42 AM Scott Sullivan < scottgregorysullivan@gmail.com> wrote:> I'm trying the following command on the source machine: > > virsh migrate --live --copy-storage-all --verbose TEST qemu+ssh:// > 10.30.76.66/system > > If I ssh into the destination machine when this command is running, I can > see NBD copying data as expected, and if I wait long enough it completes > and succeeds. > > However if I ctrl+c this command above before it completes, it causes > virsh commands (like virsh list) to just hang seemingly forever on the > destination machine. > > Here's the libvirt debug entries when I run virsh list in this state: > > [root@host ~]# export LIBVIRT_DEBUG=1 > [root@host ~]# virsh list > 2018-10-02 05:14:05.053+0000: 16207: debug : virGlobalInit:359 : register > drivers > 2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:657 > : driver=0x7f70c1ddbb60 name=Test > 2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 > : registering Test as driver 0 > 2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:657 > : driver=0x7f70c1ddc660 name=ESX > 2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 > : registering ESX as driver 12018-10-02 05:14:05.054+0000: 16207: debug : > virRegisterConnectDriver:657 : driver=0x7f70c1ddd560 name=remote > > 2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 > : registering remote as driver 2 > 2018-10-02 05:14:05.054+0000: 16207: debug : > virEventRegisterDefaultImpl:280 : registering default event implementation > 2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollAddHandle:115 : > Used 0 handle slots, adding at least 10 more > 2018-10-02 05:14:05.054+0000: 16207: debug : > virEventPollInterruptLocked:722 : Skip interrupt, 0 0 > > 2018-10-02 05:14:05.054+0000: 16207: info : virEventPollAddHandle:140 : > EVENT_POLL_ADD_HANDLE: watch=1 fd=4 events=1 cb=0x7f70c18791c0 opaque=(nil) > ff=(nil) > 2018-10-02 05:14:05.054+0000: 16207: debug : virEventRegisterImpl:241 : > addHandle=0x7f70c187a8a0 updateHandle=0x7f70c1879500 > removeHandle=0x7f70c1879360 addTimeout=0x7f70c187a660 > updateTimeout=0x7f70c1879690 removeTimeout=0x7f70c1879220 > 2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollAddTimeout:230 : > Used 0 timeout slots, adding at least 10 more2018-10-02 05:14:05.054+0000: > 16207: debug : virEventPollInterruptLocked:722 : Skip interrupt, 0 0 > 2018-10-02 05:14:05.054+0000: 16207: info : virEventPollAddTimeout:253 : > EVENT_POLL_ADD_TIMEOUT: timer=1 frequency=-1 cb=0x5559f8dba380 > opaque=0x7ffe5b15e700 ff=(nil) > 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenAuth:1218 : > name=<null>, auth=0x7f70c1ddea00, flags=0x0 > 2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: > obj=0x5559fa83a3f0 classname=virConnect > 2018-10-02 05:14:05.054+0000: 16208: debug : > virThreadJobSet:99 : Thread 16208 is now running job vshEventLoop > 2018-10-02 05:14:05.054+0000: 16208: debug : virEventRunDefaultImpl:324 : > running default event implementation > 2018-10-02 05:14:05.054+0000: 16207: debug : virConfLoadConfig:1576 : > Loading config file '/etc/libvirt/libvirt.conf' > 2018-10-02 05:14:05.054+0000: 16208: debug : > virEventPollCleanupTimeouts:525 : Cleanup 1 > 2018-10-02 05:14:05.054+0000: 16208: debug : > virEventPollCleanupHandles:574 : Cleanup 1 > 2018-10-02 05:14:05.054+0000: 16207: debug : virConfReadFile:752 : > filename=/etc/libvirt/libvirt.conf > 2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollMakePollFDs:401 : > Prepare n=0 w=1, f=4 e=1 d=0 > 2018-10-02 05:14:05.054+0000: 16208: debug : > virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers > 2018-10-02 05:14:05.054+0000: 16208: debug : > virEventPollCalculateTimeout:371 : No timeout is pending > 2018-10-02 05:14:05.054+0000: 16208: info : virEventPollRunOnce:640 : > EVENT_POLL_RUN: nhandles=1 timeout=-1 > 2018-10-02 05:14:05.054+0000: 16207: debug : virFileClose:110 : Closed fd 6 > 2018-10-02 05:14:05.054+0000: 16207: debug : virConfGetValueString:897 : > Get value string (nil) 0 > 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1013 : > no name, allowing driver auto-select > 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 : > trying driver 0 (Test) ... > 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1071 : > driver 0 Test returned DECLINED > 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 : > trying driver 1 (ESX) ... > 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1071 : > driver 1 ESX returned DECLINED > 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 : > trying driver 2 (remote) ... > 2018-10-02 05:14:05.054+0000: 16207: debug : remoteConnectOpen:1350 : > Auto-probe remote URI > 2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:916 : proceeding > with name > 2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:925 : Connecting > with transport 1 > 2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:1060 : > Proceeding with sockname /var/run/libvirt/libvirt-sock > 2018-10-02 05:14:05.054+0000: 16207: debug : > virNetSocketNewConnectUNIX:641 : path=/var/run/libvirt/libvirt-sock > spawnDaemon=0 binary=<null> > 2018-10-02 05:14:05.054+0000: 16207: debug : > virNetSocketNewConnectUNIX:705 : connect() succeeded > 2018-10-02 05:14:05.054+0000: 16207: debug : virNetSocketNew:237 : > localAddr=0x7ffe5b15dd80 remoteAddr=0x7ffe5b15dcf0 fd=6 errfd=-1 pid=0 > 2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: > obj=0x5559fa83aa20 classname=virNetSocket > 2018-10-02 05:14:05.054+0000: 16207: info : virNetSocketNew:293 : > RPC_SOCKET_NEW: sock=0x5559fa83aa20 fd=6 errfd=-1 pid=0 > localAddr=127.0.0.1;0, remoteAddr=127.0.0.1;0 > 2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: > obj=0x5559fa83ae20 classname=virNetClient > 2018-10-02 05:14:05.055+0000: 16207: info : virNetClientNew:328 : > RPC_CLIENT_NEW: client=0x5559fa83ae20 sock=0x5559fa83aa20 > 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: > obj=0x5559fa83ae20 > 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: > obj=0x5559fa83aa20 > 2018-10-02 05:14:05.055+0000: 16207: debug : > virEventPollInterruptLocked:726 : Interrupting > 2018-10-02 05:14:05.055+0000: 16207: info : virEventPollAddHandle:140 : > EVENT_POLL_ADD_HANDLE: watch=2 fd=6 events=1 cb=0x7f70c19b4df0 > opaque=0x5559fa83aa20 ff=0x7f70c19b5320 > 2018-10-02 05:14:05.055+0000: 16207: debug : virKeepAliveNew:196 : > client=0x5559fa83ae20, interval=-1, count=0 > 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: > obj=0x5559fa83b120 classname=virKeepAlive > 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollRunOnce:650 : > Poll got 1 event(s) > 2018-10-02 05:14:05.055+0000: 16207: info : virKeepAliveNew:215 : > RPC_KEEPALIVE_NEW: ka=0x5559fa83b120 client=0x5559fa83ae20 > 2018-10-02 05:14:05.055+0000: 16208: debug : > virEventPollDispatchTimeouts:432 : Dispatch 1 > 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: > obj=0x5559fa83ae20 > 2018-10-02 05:14:05.055+0000: 16208: debug : > virEventPollDispatchHandles:478 : Dispatch 1 > 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: > obj=0x5559fa83b340 classname=virConnectCloseCallbackData > 2018-10-02 05:14:05.055+0000: 16208: debug : > virEventPollDispatchHandles:492 : i=0 w=1 > 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: > obj=0x5559fa83b340 > 2018-10-02 05:14:05.055+0000: 16208: info : > virEventPollDispatchHandles:506 : EVENT_POLL_DISPATCH_HANDLE: watch=1 > events=1 > 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: > obj=0x5559fa83b040 classname=virNetClientProgram > 2018-10-02 05:14:05.055+0000: 16208: debug : > virEventPollCleanupTimeouts:525 : Cleanup 1 > 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: > obj=0x5559fa83ac00 classname=virNetClientProgram > 2018-10-02 05:14:05.055+0000: 16208: debug : > virEventPollCleanupHandles:574 : Cleanup 2 > 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: > obj=0x5559fa83a9b0 classname=virNetClientProgram > 2018-10-02 05:14:05.055+0000: 16208: debug : virEventRunDefaultImpl:324 : > running default event implementation > 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: > obj=0x5559fa83b040 > 2018-10-02 05:14:05.055+0000: 16208: debug : > virEventPollCleanupTimeouts:525 : Cleanup 1 > 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: > obj=0x5559fa83ac00 > 2018-10-02 05:14:05.055+0000: 16208: debug : > virEventPollCleanupHandles:574 : Cleanup 2 > 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: > obj=0x5559fa83a9b0 > 2018-10-02 05:14:05.055+0000: 16207: debug : doRemoteOpen:1179 : Trying > authentication > 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : > Prepare n=0 w=1, f=4 e=1 d=0 > 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : > Prepare n=1 w=2, f=6 e=1 d=0 > 2018-10-02 05:14:05.055+0000: 16207: debug : virNetMessageNew:46 : > msg=0x5559fa83b720 tracked=0 > 2018-10-02 05:14:05.055+0000: 16208: debug : > virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers > 2018-10-02 05:14:05.055+0000: 16208: debug : > virEventPollCalculateTimeout:371 : No timeout is pending > 2018-10-02 05:14:05.055+0000: 16207: debug : > virNetMessageEncodePayload:389 : Encode length as 28 > 2018-10-02 05:14:05.055+0000: 16208: info : virEventPollRunOnce:640 : > EVENT_POLL_RUN: nhandles=2 timeout=-1 > 2018-10-02 05:14:05.055+0000: 16207: info : virNetClientSendInternal:2121 > : RPC_CLIENT_MSG_TX_QUEUE: client=0x5559fa83ae20 len=28 prog=536903814 > vers=1 proc=66 type=0 status=0 serial=0 > 2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientCallNew:2074 : > New call 0x5559fa83b6a0: msg=0x5559fa83b720, expectReply=1, nonBlock=0 > 2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientIO:1880 : > Outgoing message prog=536903814 version=1 serial=0 proc=66 type=0 length=28 > dispatch=(nil) > 2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientIO:1939 : We have > the buck head=0x5559fa83b6a0 call=0x5559fa83b6a0 > 2018-10-02 05:14:05.055+0000: 16207: info : virEventPollUpdateHandle:152 : > EVENT_POLL_UPDATE_HANDLE: watch=2 events=0 > 2018-10-02 05:14:05.055+0000: 16207: debug : > virEventPollInterruptLocked:726 : Interrupting > 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollRunOnce:650 : > Poll got 1 event(s) > 2018-10-02 05:14:05.055+0000: 16208: debug : > virEventPollDispatchTimeouts:432 : Dispatch 1 > 2018-10-02 05:14:05.055+0000: 16208: debug : > virEventPollDispatchHandles:478 : Dispatch 2 > 2018-10-02 05:14:05.055+0000: 16208: debug : > virEventPollDispatchHandles:492 : i=0 w=1 > 2018-10-02 05:14:05.055+0000: 16208: info : > virEventPollDispatchHandles:506 : EVENT_POLL_DISPATCH_HANDLE: watch=1 > events=1 > 2018-10-02 05:14:05.055+0000: 16208: debug : > virEventPollCleanupTimeouts:525 : Cleanup 1 > 2018-10-02 05:14:05.055+0000: 16208: debug : > virEventPollCleanupHandles:574 : Cleanup 2 > 2018-10-02 05:14:05.055+0000: 16208: debug : virEventRunDefaultImpl:324 : > running default event implementation > 2018-10-02 05:14:05.055+0000: 16208: debug : > virEventPollCleanupTimeouts:525 : Cleanup 1 > 2018-10-02 05:14:05.055+0000: 16208: debug : > virEventPollCleanupHandles:574 : Cleanup 2 > 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : > Prepare n=0 w=1, f=4 e=1 d=0 > 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : > Prepare n=1 w=2, f=6 e=0 d=0 > 2018-10-02 05:14:05.055+0000: 16208: debug : > virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers > 2018-10-02 05:14:05.055+0000: 16208: debug : > virEventPollCalculateTimeout:371 : No timeout is pending > 2018-10-02 05:14:05.055+0000: 16208: info : virEventPollRunOnce:640 : > EVENT_POLL_RUN: nhandles=1 timeout=-1 > > And here's some information from gbd when its in this state: > > [root@host ~]# ps axf | grep libvirtd > 17010 pts/2 S+ 0:00 \_ grep libvirtd > 15555 ? Sl 0:00 libvirtd --daemon > [root@host ~]# gdb -p $(pgrep libvirtd) > > (gdb) t a a bt > > > Thread 17 (Thread 0x2ae648a10700 (LWP 15559)): > > #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value > optimized out>) at util/virthread.c:154 > #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized > out>) at util/virthreadpool.c:124 > #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at > util/virthread.c:206 > #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 > #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 > > > Thread 16 (Thread 0x2ae648c11700 (LWP 15560)): > > #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value > optimized out>) at util/virthread.c:154 > #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized > out>) at util/virthreadpool.c:124 > #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at > util/virthread.c:206 > ---Type <return> to continue, or q <return> to quit--- > #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 > > Thread 15 (Thread 0x2ae648e12700 (LWP 15561)): > #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value > optimized out>) at util/virthread.c:154 > #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized > out>) at util/virthreadpool.c:124 > #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at > util/virthread.c:206 > #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 > #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 > > Thread 14 (Thread 0x2ae649013700 (LWP 15562)): > #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value > optimized out>) at util/virthread.c:154 > ---Type <return> to continue, or q <return> to quit--- > #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized > out>) at util/virthreadpool.c:124 > #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at > util/virthread.c:206 > #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 > #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 > > > Thread 13 (Thread 0x2ae649214700 (LWP 15563)): > #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value > optimized out>) at util/virthread.c:154 > #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized > out>) at util/virthreadpool.c:124 > #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at > util/virthread.c:206 > #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 > > > Thread 12 (Thread 0x2ae649415700 (LWP 15564)): > ---Type <return> to continue, or q <return> to quit--- > > #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value > optimized out>) at util/virthread.c:154 > #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized > out>) at util/virthreadpool.c:124 > #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at > util/virthread.c:206 > #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 > #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 > > > Thread 11 (Thread 0x2ae649616700 (LWP 15565)): > > #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value > optimized out>) at util/virthread.c:154 > #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized > out>) at util/virthreadpool.c:124 > #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at > util/virthread.c:206 > #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 > #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 > > ---Type <return> to continue, or q <return> to quit--- > > > Thread 10 (Thread 0x2ae649817700 (LWP 15566)): > #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value > optimized out>) at util/virthread.c:154 > #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized > out>) at util/virthreadpool.c:124 > #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at > util/virthread.c:206 > #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 > > > Thread 9 (Thread 0x2ae649a18700 (LWP 15567)): > #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value > optimized out>) at util/virthread.c:154 > #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized > out>) at util/virthreadpool.c:124 > #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at > util/virthread.c:206 > ---Type <return> to continue, or q <return> to quit--- > > #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 > #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 > > > Thread 8 (Thread 0x2ae649c19700 (LWP 15568)): > > #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value > optimized out>) at util/virthread.c:154 > #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized > out>) at util/virthreadpool.c:124 > #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at > util/virthread.c:206 > #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 > #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 > > > Thread 7 (Thread 0x2ae64d0aa700 (LWP 15588)): > > #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value > optimized out>) at util/virthread.c:154 > ---Type <return> to continue, or q <return> to quit--- > #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized > out>) at util/virthreadpool.c:124 > #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at > util/virthread.c:206 > #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 > > > Thread 6 (Thread 0x2ae64d2ab700 (LWP 15589)): > #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value > optimized out>) at util/virthread.c:154 > #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized > out>) at util/virthreadpool.c:124 > #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at > util/virthread.c:206 > #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 > > #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 > > > Thread 5 (Thread 0x2ae64d4ac700 (LWP 15590)): > ---Type <return> to continue, or q <return> to quit--- > > #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value > optimized out>) at util/virthread.c:154 > #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized > out>) at util/virthreadpool.c:124 > #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at > util/virthread.c:206 > #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 > #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 > > > Thread 4 (Thread 0x2ae64d6ad700 (LWP 15591)): > > #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value > optimized out>) at util/virthread.c:154 > #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized > out>) at util/virthreadpool.c:124 > #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at > util/virthread.c:206 > #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 > #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 > > ---Type <return> to continue, or q <return> to quit--- > > > Thread 3 (Thread 0x2ae64d8ae700 (LWP 15592)): > #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value > optimized out>) at util/virthread.c:154 > #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized > out>) at util/virthreadpool.c:124 > #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at > util/virthread.c:206 > #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 > #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 > > Thread 2 (Thread 0x2ae64dec8700 (LWP 15632)): > #0 0x00002ae63d97ed0d in recvmsg () from /lib64/libpthread.so.0 > #1 0x00002ae63e9caef0 in udev_monitor_receive_device () from > /lib64/libudev.so.0 > #2 0x00002ae64be26864 in udevEventHandleThread (opaque=<value optimized > out>) at node_device/node_device_udev.c:1623 > #3 0x00002ae63b2c0075 in virThreadHelper (data=<value optimized out>) at > util/virthread.c:206 > ---Type <return> to continue, or q <return> to quit--- > #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 > #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 > > Thread 1 (Thread 0x2ae6429759e0 (LWP 15555)): > #0 0x00002ae63d97e334 in __lll_lock_wait () from /lib64/libpthread.so.0 > #1 0x00002ae63d9795d8 in _L_lock_854 () from /lib64/libpthread.so.0 > #2 0x00002ae63d9794a7 in pthread_mutex_lock () from /lib64/libpthread.so.0 > #3 0x00002ae64be2364b in udevEventHandleCallback (watch=<value optimized > out>, fd=19, events=<value optimized out>, data=<value optimized out>) at > node_device/node_device_udev.c:1666 > #4 0x00002ae63b257564 in virEventPollDispatchHandles () at > util/vireventpoll.c:508 > #5 virEventPollRunOnce () at util/vireventpoll.c:657 > #6 0x00002ae63b255eb0 in virEventRunDefaultImpl () at util/virevent.c:327 > #7 0x00002ae63b3a338d in virNetDaemonRun (dmn=0x55b1d39818d0) at > rpc/virnetdaemon.c:858 > #8 0x000055b1d2a4d60b in main (argc=<value optimized out>, argv=<value > optimized out>) at remote/remote_daemon.c:1475 > (gdb) > > This issue I know is fixed in 4.7.0 because I run this version on my > CentOS 7 machines and don't have this issue. However on my CentOS 6 > machines I have 4.2.0, and do experience this issue. Upgrading to 4.7.0 on > CentOS 6 is troublesome though, because of the requirement of gnutls >= 3.2. > > I'm hoping with the above output someone might be able to point me in the > right direction to a commit that resolved this between 4.2.0 and 4.7.0, or > a possible way to prevent this hang in 4.2.0 as described above ? > > Thanks in advance >