thr3ads.net - libvirt users - Re: [libvirt-users] libvirt hang on domain attemp to create domain [Jun 2013]

If this information is useful, please help other people find it:
Share via:

Александр

2013-Jun-19 19:24 UTC

Re: [libvirt-users] libvirt hang on domain attemp to create domain

В письме от Среда, 19-июн-2013 09:35:27 пользователь Michal Privoznik
написал:> On 18.06.2013 22:21, Александр wrote:
> > В письме от Вторник, 18-июн-2013 12:10:41 пользователь Michal
Privoznik
> > 
> > написал:
> >> On 18.06.2013 07:14, Александр wrote:
> >>> good day, i am trying to create domain via virDomainDefineXML.
and my
> >>> program hanging, also i failed to caonnect to libvirt daemon
with
> >>> virt-manager after this, only killing libvirtd via SIGKILL and
> >>> restarting
> >>> it solveing hang (i mean after restart i able to connect with
> >>> virt-manager), i using following domain xml
> >>> 
> >>> <?xml version="1.0"
encoding="UTF-8"?>
> >>> <domain type="kvm">
> >>> 
> >>>   <tittle>Automatically created domain with default
> >>>   configuration</tittle>
> >>>   <name>vm1</name>
> >>>   <description>no description</description>
> >>>   <os>
> >>>   
> >>>     <type>hvm</type>
> >>>     <boot dev="cdrom"/>
> >>>     <boot dev="hd"/>
> >>>   
> >>>   </os>
> >>>   <vcpu>1</vcpu>
> >>>   <memory unit="MB">512</memory>
> >>>   <cpu model="host-model">
> >>>   
> >>>     <topology sockets="1" cores="1"
threads="1"/>
> >>>   
> >>>   </cpu>
> >>>   <clock offset="localtime"/>
> >>>   <devices>
> >>>   
> >>>     <graphics type="vnc" autoport="yes"
password="admin"/>
> >>>     <channel type="unix">
> >>>     
> >>>       <source mode="bind"
path="/var/run/vm1.sock"/>
> >>>       <target type="virtio"
name="org.guest_agent.0"/>
> >>>     
> >>>     </channel>
> >>>   
> >>>   </devices>
> >>> 
> >>> </domain>
> >>> 
> >>> i have this calls on stack
> >>> 
> >>> 0  0x00007ffff48c1750  __poll_nocancel
> >>> 1  0x00007ffff649e5ca  virNetClientIOEventLoop
> >> 
> >> This is just probably a client waiting for the corresponding
libvirtd to
> >> provide a reply message. Are you able to get libvirtd debug logs
so we
> >> can see what's going on?
> >> 
> >> http://wiki.libvirt.org/page/DebugLogs
> >> 
> >> BTW, what's the libvirt version you are running?
> >> 
> >> Michal
> >> 
> >>> 2  0x00007ffff649ed96  virNetClientSendInternal
> >>> 3  0x00007ffff64a0443  virNetClientSendWithReply
> >>> 4  0x00007ffff64a0daa  virNetClientProgramCall
> >>> 5  0x00007ffff6478566  callFull.isra.2
> >>> 6  0x00007ffff6487fa2  remoteDomainDefineXML
> >>> 7  0x00007ffff6449d2f  virDomainDefineXML
> >>> 8  0x0000000000426941  virt_domain::create
> >>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp 
295
> >>> 9  0x000000000040c1a9  main
> >>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp  40
> >>> 
> >>> what i am doing wrong ?
> >>> 
> >>> 
> >>> 
> >>> _______________________________________________
> >>> libvirt-users mailing list
> >>> libvirt-users@redhat.com
> >>> https://www.redhat.com/mailman/listinfo/libvirt-users
> > 
> > i have libvirt version 1.0.6, debug log in attachment.
> 
> Yuck, this is 1.0.6; Well, from the logs it seems like the daemon is
> stuck. Can you attach to the libvirtd while it's being stuck and run
't
> a a bt' and post the result? I wonder if we have a deadlock somewhere.
> 
> Michal
if i correctly understand you, you need debugger backtrace after libvirtd 
stuck ?, if so, here is bt from gdb

(gdb) bt
#0  0x00007f00c1ae876d in poll () from /lib64/libc.so.6
#1  0x00007f00c203cfcb in virEventPollRunOnce () from /usr/lib64/libvirt.so.0
#2  0x00007f00c203ba4d in virEventRunDefaultImpl () from 
/usr/lib64/libvirt.so.0
#3  0x00007f00c214a5cd in virNetServerRun () from /usr/lib64/libvirt.so.0
#4  0x00007f00c2aea08a in main ()


also maybe build options means..., here is my build options

* app-emulation/libvirt
    ::gentoo                  1.0.3-r2 1.0.5.1-r3 1.0.5.2(~) 1.0.6(~)* (9999)K 
{:0}
    ::installed               1.0.6 {:0}
    app-emulation/libvirt-1.0.6:0::installed (world)
    Description               C toolkit to manipulate virtual machines
    Homepage                  http://www.libvirt.org/
    From repositories         gentoo
    Installed time            Tue Jun 18 07:18:08 EEST 2013
    Installed using           paludis-1.4.0
    License                   LGPL-2.1
    Use flags                 
        USE                   
            (-audit)          
            (-avahi)          
            (caps)            
            (-firewalld)      
            (fuse)            
            (iscsi)           
            (libvirtd)        
            (lvm)             
            (-lxc)            
            (macvtap)         
            (nfs)             
            (nls)             
            (-numa)           
            (-openvz)         
            (-parted)         
            (pcap)            
            (-phyp)           
            (policykit)       
            (python)          
            (qemu)            
            (-rbd)            
            (sasl)            
            (-selinux)        
            (-systemd)        
            (udev)            
            (-uml)            
            (vepa)            
            (-virt-network)   
            (virtualbox)      
            (-xen)            
        python_single_target  
            (-python2_5)      
            (-python2_6)      
            (python2_7)       
        python_targets        
            (-python2_5)      
            (-python2_6)      
            (python2_7)       
        Build Options         
            -trace            Trace actions executed by the package (very 
noisy, for debugging broken builds only)

Michal Privoznik

2013-Jun-20 07:15 UTC

head link

Re: [libvirt-users] libvirt hang on domain attemp to create domain

On 19.06.2013 21:24, Александр wrote:> В письме от Среда, 19-июн-2013 09:35:27 пользователь Michal Privoznik
написал:
>> On 18.06.2013 22:21, Александр wrote:
>>> В письме от Вторник, 18-июн-2013 12:10:41 пользователь Michal
Privoznik
>>>
>>> написал:
>>>> On 18.06.2013 07:14, Александр wrote:
>>>>> good day, i am trying to create domain via
virDomainDefineXML. and my
>>>>> program hanging, also i failed to caonnect to libvirt
daemon with
>>>>> virt-manager after this, only killing libvirtd via SIGKILL
and
>>>>> restarting
>>>>> it solveing hang (i mean after restart i able to connect
with
>>>>> virt-manager), i using following domain xml
>>>>>
>>>>> <?xml version="1.0"
encoding="UTF-8"?>
>>>>> <domain type="kvm">
>>>>>
>>>>>   <tittle>Automatically created domain with default
>>>>>   configuration</tittle>
>>>>>   <name>vm1</name>
>>>>>   <description>no description</description>
>>>>>   <os>
>>>>>   
>>>>>     <type>hvm</type>
>>>>>     <boot dev="cdrom"/>
>>>>>     <boot dev="hd"/>
>>>>>   
>>>>>   </os>
>>>>>   <vcpu>1</vcpu>
>>>>>   <memory unit="MB">512</memory>
>>>>>   <cpu model="host-model">
>>>>>   
>>>>>     <topology sockets="1" cores="1"
threads="1"/>
>>>>>   
>>>>>   </cpu>
>>>>>   <clock offset="localtime"/>
>>>>>   <devices>
>>>>>   
>>>>>     <graphics type="vnc"
autoport="yes" password="admin"/>
>>>>>     <channel type="unix">
>>>>>     
>>>>>       <source mode="bind"
path="/var/run/vm1.sock"/>
>>>>>       <target type="virtio"
name="org.guest_agent.0"/>
>>>>>     
>>>>>     </channel>
>>>>>   
>>>>>   </devices>
>>>>>
>>>>> </domain>
>>>>>
>>>>> i have this calls on stack
>>>>>
>>>>> 0  0x00007ffff48c1750  __poll_nocancel
>>>>> 1  0x00007ffff649e5ca  virNetClientIOEventLoop
>>>>
>>>> This is just probably a client waiting for the corresponding
libvirtd to
>>>> provide a reply message. Are you able to get libvirtd debug
logs so we
>>>> can see what's going on?
>>>>
>>>> http://wiki.libvirt.org/page/DebugLogs
>>>>
>>>> BTW, what's the libvirt version you are running?
>>>>
>>>> Michal
>>>>
>>>>> 2  0x00007ffff649ed96  virNetClientSendInternal
>>>>> 3  0x00007ffff64a0443  virNetClientSendWithReply
>>>>> 4  0x00007ffff64a0daa  virNetClientProgramCall
>>>>> 5  0x00007ffff6478566  callFull.isra.2
>>>>> 6  0x00007ffff6487fa2  remoteDomainDefineXML
>>>>> 7  0x00007ffff6449d2f  virDomainDefineXML
>>>>> 8  0x0000000000426941  virt_domain::create
>>>>>
/home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp  295
>>>>> 9  0x000000000040c1a9  main
>>>>> /home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp 
40
>>>>>
>>>>> what i am doing wrong ?
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> libvirt-users mailing list
>>>>> libvirt-users@redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/libvirt-users
>>>
>>> i have libvirt version 1.0.6, debug log in attachment.
>>
>> Yuck, this is 1.0.6; Well, from the logs it seems like the daemon is
>> stuck. Can you attach to the libvirtd while it's being stuck and
run 't
>> a a bt' and post the result? I wonder if we have a deadlock
somewhere.
>>
>> Michal
> 
> if i correctly understand you, you need debugger backtrace after libvirtd 
> stuck ?, if so, here is bt from gdb
You understand correctly. However, simple 'bt' doesn't show
what's going
on in other threads - libvirtd is multi threaded.
> 
> (gdb) bt
> #0  0x00007f00c1ae876d in poll () from /lib64/libc.so.6
> #1  0x00007f00c203cfcb in virEventPollRunOnce () from
/usr/lib64/libvirt.so.0
> #2  0x00007f00c203ba4d in virEventRunDefaultImpl () from 
> /usr/lib64/libvirt.so.0
> #3  0x00007f00c214a5cd in virNetServerRun () from /usr/lib64/libvirt.so.0
> #4  0x00007f00c2aea08a in main ()
So all we see here is the libvirtd event loop. That's why we need 't a a
bt' which is 'thread apply all backtrace' which will run
'bt' over all
threads.

Michal

Alexandr

2013-Jun-20 10:56 UTC

head link

Re: [libvirt-users] libvirt hang on domain attemp to create domain

On 20.06.2013 10:44, Michal Privoznik wrote:> On 20.06.2013 09:40, Александр wrote:
>> В письме от Четверг, 20-июн-2013 09:15:14 Вы написали:
>>> On 19.06.2013 21:24, Александр wrote:
>>>> В письме от Среда, 19-июн-2013 09:35:27 пользователь Michal
Privoznik
>> написал:
>>>>> On 18.06.2013 22:21, Александр wrote:
>>>>>> В письме от Вторник, 18-июн-2013 12:10:41 пользователь
Michal Privoznik
>>>>>>
>>>>>> написал:
>>>>>>> On 18.06.2013 07:14, Александр wrote:
>>>>>>>> good day, i am trying to create domain via
virDomainDefineXML. and my
>>>>>>>> program hanging, also i failed to caonnect to
libvirt daemon with
>>>>>>>> virt-manager after this, only killing libvirtd
via SIGKILL and
>>>>>>>> restarting
>>>>>>>> it solveing hang (i mean after restart i able
to connect with
>>>>>>>> virt-manager), i using following domain xml
>>>>>>>>
>>>>>>>> <?xml version="1.0"
encoding="UTF-8"?>
>>>>>>>> <domain type="kvm">
>>>>>>>>
>>>>>>>>    <tittle>Automatically created domain
with default
>>>>>>>>    configuration</tittle>
>>>>>>>>    <name>vm1</name>
>>>>>>>>    <description>no
description</description>
>>>>>>>>    <os>
>>>>>>>>    
>>>>>>>>      <type>hvm</type>
>>>>>>>>      <boot dev="cdrom"/>
>>>>>>>>      <boot dev="hd"/>
>>>>>>>>    
>>>>>>>>    </os>
>>>>>>>>    <vcpu>1</vcpu>
>>>>>>>>    <memory
unit="MB">512</memory>
>>>>>>>>    <cpu model="host-model">
>>>>>>>>    
>>>>>>>>      <topology sockets="1"
cores="1" threads="1"/>
>>>>>>>>    
>>>>>>>>    </cpu>
>>>>>>>>    <clock offset="localtime"/>
>>>>>>>>    <devices>
>>>>>>>>    
>>>>>>>>      <graphics type="vnc"
autoport="yes" password="admin"/>
>>>>>>>>      <channel type="unix">
>>>>>>>>      
>>>>>>>>        <source mode="bind"
path="/var/run/vm1.sock"/>
>>>>>>>>        <target type="virtio"
name="org.guest_agent.0"/>
>>>>>>>>      
>>>>>>>>      </channel>
>>>>>>>>    
>>>>>>>>    </devices>
>>>>>>>>
>>>>>>>> </domain>
>>>>>>>>
>>>>>>>> i have this calls on stack
>>>>>>>>
>>>>>>>> 0  0x00007ffff48c1750  __poll_nocancel
>>>>>>>> 1  0x00007ffff649e5ca  virNetClientIOEventLoop
>>>>>>> This is just probably a client waiting for the
corresponding libvirtd
>>>>>>> to
>>>>>>> provide a reply message. Are you able to get
libvirtd debug logs so we
>>>>>>> can see what's going on?
>>>>>>>
>>>>>>> http://wiki.libvirt.org/page/DebugLogs
>>>>>>>
>>>>>>> BTW, what's the libvirt version you are
running?
>>>>>>>
>>>>>>> Michal
>>>>>>>
>>>>>>>> 2  0x00007ffff649ed96  virNetClientSendInternal
>>>>>>>> 3  0x00007ffff64a0443 
virNetClientSendWithReply
>>>>>>>> 4  0x00007ffff64a0daa  virNetClientProgramCall
>>>>>>>> 5  0x00007ffff6478566  callFull.isra.2
>>>>>>>> 6  0x00007ffff6487fa2  remoteDomainDefineXML
>>>>>>>> 7  0x00007ffff6449d2f  virDomainDefineXML
>>>>>>>> 8  0x0000000000426941  virt_domain::create
>>>>>>>>
/home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp  295
>>>>>>>> 9  0x000000000040c1a9  main
>>>>>>>>
/home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp  40
>>>>>>>>
>>>>>>>> what i am doing wrong ?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> libvirt-users mailing list
>>>>>>>> libvirt-users@redhat.com
>>>>>>>>
https://www.redhat.com/mailman/listinfo/libvirt-users
>>>>>> i have libvirt version 1.0.6, debug log in attachment.
>>>>> Yuck, this is 1.0.6; Well, from the logs it seems like the
daemon is
>>>>> stuck. Can you attach to the libvirtd while it's being
stuck and run 't
>>>>> a a bt' and post the result? I wonder if we have a
deadlock somewhere.
>>>>>
>>>>> Michal
>>>> if i correctly understand you, you need debugger backtrace
after libvirtd
>>>> stuck ?, if so, here is bt from gdb
>>> You understand correctly. However, simple 'bt' doesn't
show what's going
>>> on in other threads - libvirtd is multi threaded.
>>>
>>>> (gdb) bt
>>>> #0  0x00007f00c1ae876d in poll () from /lib64/libc.so.6
>>>> #1  0x00007f00c203cfcb in virEventPollRunOnce () from
>>>> /usr/lib64/libvirt.so.0 #2  0x00007f00c203ba4d in
virEventRunDefaultImpl
>>>> () from
>>>> /usr/lib64/libvirt.so.0
>>>> #3  0x00007f00c214a5cd in virNetServerRun () from
/usr/lib64/libvirt.so.0
>>>> #4  0x00007f00c2aea08a in main ()
>>> So all we see here is the libvirtd event loop. That's why we
need 't a a
>>> bt' which is 'thread apply all backtrace' which will
run 'bt' over all
>>> threads.
>>>
>>> Michal
>> how can i make it ?
>>
> The very same way as you did 'bt'. Just instead of typing
'bt' you'll
> type 't a a bt'.
>
> Michalhere is backtrace from all threads:

(gdb) t a a bt

Thread 11 (Thread 0x7fdd8db82700 (LWP 20321)):
#0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2  0x00007fdd9476582b in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fdd8d381700 (LWP 20322)):
#0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2  0x00007fdd9476582b in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fdd8cb80700 (LWP 20323)):
#0  0x00007fdd944bc344 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fdd944b7e56 in _L_lock_511 () from /lib64/libpthread.so.0
#2  0x00007fdd944b7caa in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007fdd94773861 in virDomainObjListSearchName () from 
/usr/lib64/libvirt.so.0
#4  0x00007fdd9473c8a5 in virHashSearch () from /usr/lib64/libvirt.so.0
#5  0x00007fdd9478cc3d in virDomainObjListAddLocked.isra.43 () from 
/usr/lib64/libvirt.so.0
#6  0x00007fdd9478cd47 in virDomainObjListAdd () from 
/usr/lib64/libvirt.so.0
#7  0x00007fdd875a311c in qemuDomainDefineXML () from 
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#8  0x00007fdd947ead2f in virDomainDefineXML () from /usr/lib64/libvirt.so.0
#9  0x00007fdd95206b4d in remoteDispatchDomainDefineXMLHelper ()
#10 0x00007fdd94848d27 in virNetServerProgramDispatch () from 
/usr/lib64/libvirt.so.0
#11 0x00007fdd94844068 in virNetServerHandleJob () from 
/usr/lib64/libvirt.so.0
#12 0x00007fdd94765795 in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#13 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#14 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#15 0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fdd8c37f700 (LWP 20324)):
#0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2  0x00007fdd9476582b in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fdd8bb7e700 (LWP 20325)):
#0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2  0x00007fdd9476582b in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fdd8b37d700 (LWP 20326)):
#0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2  0x00007fdd9476584b in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fdd8ab7c700 (LWP 20327)):
#0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2  0x00007fdd9476584b in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fdd8a37b700 (LWP 20328)):
#0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2  0x00007fdd9476584b in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fdd89b7a700 (LWP 20329)):
#0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2  0x00007fdd9476584b in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fdd89379700 (LWP 20330)):
#0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
#2  0x00007fdd9476584b in virThreadPoolWorker () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
#4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fdd95166740 (LWP 20320)):
#0  0x00007fdd941e376d in poll () from /lib64/libc.so.6
#1  0x00007fdd94737fcb in virEventPollRunOnce () from 
/usr/lib64/libvirt.so.0
#2  0x00007fdd94736a4d in virEventRunDefaultImpl () from 
/usr/lib64/libvirt.so.0
#3  0x00007fdd948455cd in virNetServerRun () from /usr/lib64/libvirt.so.0
#4  0x00007fdd951e508a in main ()

and also from my program if it important:

0  0x00007ffff48c1750  __poll_nocancel
1  0x00007ffff649e5ca  virNetClientIOEventLoop
2  0x00007ffff649ed96  virNetClientSendInternal
3  0x00007ffff64a0443  virNetClientSendWithReply
4  0x00007ffff64a0daa  virNetClientProgramCall
5  0x00007ffff6478566  callFull.isra.2
6  0x00007ffff6487fa2  remoteDomainDefineXML
7  0x00007ffff6449d2f  virDomainDefineXML
8  0x0000000000426941  virt_domain::create 
/home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp  295
9  0x000000000040c1a9  main 
/home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp  40

Michal Privoznik

2013-Jun-20 11:25 UTC

head link

Re: [libvirt-users] libvirt hang on domain attemp to create domain

On 20.06.2013 12:56, Alexandr wrote:> On 20.06.2013 10:44, Michal Privoznik wrote:
>> On 20.06.2013 09:40, Александр wrote:
>>> В письме от Четверг, 20-июн-2013 09:15:14 Вы написали:
>>>> On 19.06.2013 21:24, Александр wrote:
>>>>> В письме от Среда, 19-июн-2013 09:35:27 пользователь Michal
Privoznik
>>> написал:
>>>>>> On 18.06.2013 22:21, Александр wrote:
>>>>>>> В письме от Вторник, 18-июн-2013 12:10:41
пользователь Michal
>>>>>>> Privoznik
>>>>>>>
>>>>>>> написал:
>>>>>>>> On 18.06.2013 07:14, Александр wrote:
>>>>>>>>> good day, i am trying to create domain via
virDomainDefineXML.
>>>>>>>>> and my
>>>>>>>>> program hanging, also i failed to caonnect
to libvirt daemon with
>>>>>>>>> virt-manager after this, only killing
libvirtd via SIGKILL and
>>>>>>>>> restarting
>>>>>>>>> it solveing hang (i mean after restart i
able to connect with
>>>>>>>>> virt-manager), i using following domain xml
>>>>>>>>>
>>>>>>>>> <?xml version="1.0"
encoding="UTF-8"?>
>>>>>>>>> <domain type="kvm">
>>>>>>>>>
>>>>>>>>>    <tittle>Automatically created
domain with default
>>>>>>>>>    configuration</tittle>
>>>>>>>>>    <name>vm1</name>
>>>>>>>>>    <description>no
description</description>
>>>>>>>>>    <os>
>>>>>>>>>         <type>hvm</type>
>>>>>>>>>      <boot dev="cdrom"/>
>>>>>>>>>      <boot dev="hd"/>
>>>>>>>>>       </os>
>>>>>>>>>    <vcpu>1</vcpu>
>>>>>>>>>    <memory
unit="MB">512</memory>
>>>>>>>>>    <cpu model="host-model">
>>>>>>>>>         <topology sockets="1"
cores="1" threads="1"/>
>>>>>>>>>       </cpu>
>>>>>>>>>    <clock
offset="localtime"/>
>>>>>>>>>    <devices>
>>>>>>>>>         <graphics type="vnc"
autoport="yes" password="admin"/>
>>>>>>>>>      <channel type="unix">
>>>>>>>>>             <source
mode="bind" path="/var/run/vm1.sock"/>
>>>>>>>>>        <target type="virtio"
name="org.guest_agent.0"/>
>>>>>>>>>           </channel>
>>>>>>>>>       </devices>
>>>>>>>>>
>>>>>>>>> </domain>
>>>>>>>>>
>>>>>>>>> i have this calls on stack
>>>>>>>>>
>>>>>>>>> 0  0x00007ffff48c1750  __poll_nocancel
>>>>>>>>> 1  0x00007ffff649e5ca 
virNetClientIOEventLoop
>>>>>>>> This is just probably a client waiting for the
corresponding
>>>>>>>> libvirtd
>>>>>>>> to
>>>>>>>> provide a reply message. Are you able to get
libvirtd debug logs
>>>>>>>> so we
>>>>>>>> can see what's going on?
>>>>>>>>
>>>>>>>> http://wiki.libvirt.org/page/DebugLogs
>>>>>>>>
>>>>>>>> BTW, what's the libvirt version you are
running?
>>>>>>>>
>>>>>>>> Michal
>>>>>>>>
>>>>>>>>> 2  0x00007ffff649ed96 
virNetClientSendInternal
>>>>>>>>> 3  0x00007ffff64a0443 
virNetClientSendWithReply
>>>>>>>>> 4  0x00007ffff64a0daa 
virNetClientProgramCall
>>>>>>>>> 5  0x00007ffff6478566  callFull.isra.2
>>>>>>>>> 6  0x00007ffff6487fa2 
remoteDomainDefineXML
>>>>>>>>> 7  0x00007ffff6449d2f  virDomainDefineXML
>>>>>>>>> 8  0x0000000000426941  virt_domain::create
>>>>>>>>>
/home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp  295
>>>>>>>>> 9  0x000000000040c1a9  main
>>>>>>>>>
/home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp  40
>>>>>>>>>
>>>>>>>>> what i am doing wrong ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
_______________________________________________
>>>>>>>>> libvirt-users mailing list
>>>>>>>>> libvirt-users@redhat.com
>>>>>>>>>
https://www.redhat.com/mailman/listinfo/libvirt-users
>>>>>>> i have libvirt version 1.0.6, debug log in
attachment.
>>>>>> Yuck, this is 1.0.6; Well, from the logs it seems like
the daemon is
>>>>>> stuck. Can you attach to the libvirtd while it's
being stuck and
>>>>>> run 't
>>>>>> a a bt' and post the result? I wonder if we have a
deadlock
>>>>>> somewhere.
>>>>>>
>>>>>> Michal
>>>>> if i correctly understand you, you need debugger backtrace
after
>>>>> libvirtd
>>>>> stuck ?, if so, here is bt from gdb
>>>> You understand correctly. However, simple 'bt'
doesn't show what's
>>>> going
>>>> on in other threads - libvirtd is multi threaded.
>>>>
>>>>> (gdb) bt
>>>>> #0  0x00007f00c1ae876d in poll () from /lib64/libc.so.6
>>>>> #1  0x00007f00c203cfcb in virEventPollRunOnce () from
>>>>> /usr/lib64/libvirt.so.0 #2  0x00007f00c203ba4d in
>>>>> virEventRunDefaultImpl
>>>>> () from
>>>>> /usr/lib64/libvirt.so.0
>>>>> #3  0x00007f00c214a5cd in virNetServerRun () from
>>>>> /usr/lib64/libvirt.so.0
>>>>> #4  0x00007f00c2aea08a in main ()
>>>> So all we see here is the libvirtd event loop. That's why
we need 't
>>>> a a
>>>> bt' which is 'thread apply all backtrace' which
will run 'bt' over all
>>>> threads.
>>>>
>>>> Michal
>>> how can i make it ?
>>>
>> The very same way as you did 'bt'. Just instead of typing
'bt' you'll
>> type 't a a bt'.
>>
>> Michal
> here is backtrace from all threads:
> 
> (gdb) t a a bt
> 
> Thread 11 (Thread 0x7fdd8db82700 (LWP 20321)):
> #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> #2  0x00007fdd9476582b in virThreadPoolWorker () from
> /usr/lib64/libvirt.so.0
> #3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> 
> Thread 10 (Thread 0x7fdd8d381700 (LWP 20322)):
> #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> #2  0x00007fdd9476582b in virThreadPoolWorker () from
> /usr/lib64/libvirt.so.0
> #3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> 
> Thread 9 (Thread 0x7fdd8cb80700 (LWP 20323)):
> #0  0x00007fdd944bc344 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1  0x00007fdd944b7e56 in _L_lock_511 () from /lib64/libpthread.so.0
> #2  0x00007fdd944b7caa in pthread_mutex_lock () from /lib64/libpthread.so.0
Bingo! So we've a deadlock here. Somebody has forgotten to unlock a
domain. So as we go over the domain list to check for name (if it exists
already or not), we deadlock. The process goes like this:

1) if adding a domain the the internal list (I should write rather hash
table where UUID is the key, and the pointer to the internal structure
representing domain is the value), we search for UUID. Since this is the
key, we don't have to lock anything (besides the hash table itself).

2) Okay, no matching UUID. But what about domain name? We go through the
table again, over each element, and:
2a) lock the element
2b) compare the names
2c) unlock the element

And since a domain withing the table is locked (and all none of the
threads is using it) we deadlock in step 2a) as there's no one to come
and unlock the domain. Sigh. Hopefully, I'll have some time to dig into
the log you've provided and get more precise info.
> #3  0x00007fdd94773861 in virDomainObjListSearchName () from
> /usr/lib64/libvirt.so.0
> #4  0x00007fdd9473c8a5 in virHashSearch () from /usr/lib64/libvirt.so.0
> #5  0x00007fdd9478cc3d in virDomainObjListAddLocked.isra.43 () from
> /usr/lib64/libvirt.so.0
> #6  0x00007fdd9478cd47 in virDomainObjListAdd () from
> /usr/lib64/libvirt.so.0
> #7  0x00007fdd875a311c in qemuDomainDefineXML () from
> /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
> #8  0x00007fdd947ead2f in virDomainDefineXML () from
> /usr/lib64/libvirt.so.0
> #9  0x00007fdd95206b4d in remoteDispatchDomainDefineXMLHelper ()
> #10 0x00007fdd94848d27 in virNetServerProgramDispatch () from
> /usr/lib64/libvirt.so.0
> #11 0x00007fdd94844068 in virNetServerHandleJob () from
> /usr/lib64/libvirt.so.0
> #12 0x00007fdd94765795 in virThreadPoolWorker () from
> /usr/lib64/libvirt.so.0
> #13 0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> #14 0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> #15 0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> 
> Thread 8 (Thread 0x7fdd8c37f700 (LWP 20324)):
> #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> #2  0x00007fdd9476582b in virThreadPoolWorker () from
> /usr/lib64/libvirt.so.0
> #3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> 
> Thread 7 (Thread 0x7fdd8bb7e700 (LWP 20325)):
> #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> #2  0x00007fdd9476582b in virThreadPoolWorker () from
> /usr/lib64/libvirt.so.0
> #3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> 
> Thread 6 (Thread 0x7fdd8b37d700 (LWP 20326)):
> #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> #2  0x00007fdd9476584b in virThreadPoolWorker () from
> /usr/lib64/libvirt.so.0
> #3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> 
> Thread 5 (Thread 0x7fdd8ab7c700 (LWP 20327)):
> #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> #2  0x00007fdd9476584b in virThreadPoolWorker () from
> /usr/lib64/libvirt.so.0
> #3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> 
> Thread 4 (Thread 0x7fdd8a37b700 (LWP 20328)):
> #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> ---Type <return> to continue, or q <return> to quit---
> #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> #2  0x00007fdd9476584b in virThreadPoolWorker () from
> /usr/lib64/libvirt.so.0
> #3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> 
> Thread 3 (Thread 0x7fdd89b7a700 (LWP 20329)):
> #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> #2  0x00007fdd9476584b in virThreadPoolWorker () from
> /usr/lib64/libvirt.so.0
> #3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> 
> Thread 2 (Thread 0x7fdd89379700 (LWP 20330)):
> #0  0x00007fdd944b990c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00007fdd9476515a in virCondWait () from /usr/lib64/libvirt.so.0
> #2  0x00007fdd9476584b in virThreadPoolWorker () from
> /usr/lib64/libvirt.so.0
> #3  0x00007fdd94764dfe in virThreadHelper () from /usr/lib64/libvirt.so.0
> #4  0x00007fdd944b5d63 in start_thread () from /lib64/libpthread.so.0
> #5  0x00007fdd941ed04d in clone () from /lib64/libc.so.6
> 
> Thread 1 (Thread 0x7fdd95166740 (LWP 20320)):
> #0  0x00007fdd941e376d in poll () from /lib64/libc.so.6
> #1  0x00007fdd94737fcb in virEventPollRunOnce () from
> /usr/lib64/libvirt.so.0
> #2  0x00007fdd94736a4d in virEventRunDefaultImpl () from
> /usr/lib64/libvirt.so.0
> #3  0x00007fdd948455cd in virNetServerRun () from /usr/lib64/libvirt.so.0
> #4  0x00007fdd951e508a in main ()
> 
> and also from my program if it important:
> 
> 0  0x00007ffff48c1750  __poll_nocancel
> 1  0x00007ffff649e5ca  virNetClientIOEventLoop
> 2  0x00007ffff649ed96  virNetClientSendInternal
> 3  0x00007ffff64a0443  virNetClientSendWithReply
> 4  0x00007ffff64a0daa  virNetClientProgramCall
> 5  0x00007ffff6478566  callFull.isra.2
> 6  0x00007ffff6487fa2  remoteDomainDefineXML
> 7  0x00007ffff6449d2f  virDomainDefineXML
> 8  0x0000000000426941  virt_domain::create
> /home/sss/install/git/work/avirt/vm_pool/vm_pool/libvirt.cpp  295
> 9  0x000000000040c1a9  main
> /home/sss/install/git/work/avirt/vm_pool/vm_pool/main.cpp  40
> 
Michal

Reasonably Related Threads

Search for more possibly parallel threads

libvirt users - Jun 2013 - Re: libvirt hang on domain attemp to create domain

Re: [libvirt-users] libvirt hang on domain attemp to create domain

Re: [libvirt-users] libvirt hang on domain attemp to create domain

Re: [libvirt-users] libvirt hang on domain attemp to create domain

Re: [libvirt-users] libvirt hang on domain attemp to create domain

Reasonably Related Threads