libvirt users - May 2013 - hangs in libvirt function virLogMessage

Hi,
  We are hitting a deadlock on libvirt on 0.9.4 version. Can someone help us ? 

1) Is there any way to specify timeout for all the calls made by libvirt
especially virConnectOpenReadOnly

2) Does virCommandRunAsync always fork a new process ?


gdb) bt
#0  0x00007fc17b53ce6e in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x00007fc17b4e3f2d in _L_lock_2164 () from /lib64/libc.so.6
#2  0x00007fc17b4e3ce7 in __tz_convert () from /lib64/libc.so.6
#3  0x00007fc16e4bfdab in virLogMessage () from /usr/lib64/libvirt.so.0
#4  0x00007fc16e4b092e in ?? () from /usr/lib64/libvirt.so.0
#5  0x00007fc16e4b2836 in ?? () from /usr/lib64/libvirt.so.0
#6  0x00007fc16e4b318c in virCommandRunAsync () from /usr/lib64/libvirt.so.0
#7  0x00007fc16e570705 in ?? () from /usr/lib64/libvirt.so.0
#8  0x00007fc16e56a1f8 in ?? () from /usr/lib64/libvirt.so.0
#9  0x00007fc16e55c3d2 in ?? () from /usr/lib64/libvirt.so.0
#10 0x00007fc16e55db10 in ?? () from /usr/lib64/libvirt.so.0
#11 0x00007fc16e52ed95 in ?? () from /usr/lib64/libvirt.so.0
#12 0x00007fc16e52f957 in virConnectOpenReadOnly () from /usr/lib64/libvirt.so.0
#13 0x00007fc153cdb858 in OalibvirtCollectionManager::libvirtColJobImpl::Run
(this=0x7fc13182a610)
    at ../libvirtColJobImpl.cpp:348
#14 0x00007fc153cd6e83 in OalibvirtCollectionManager::libvirtColJob::Run
(this=<value optimized out>)
    at ../libvirtColJob.cpp:36
#15 0x00007fc17cc8ee36 in OAScheduler::ScheduleWorker::Run (this=0x7fc15dd9ede0,
    myThread=<value optimized out>)
    at
/svn-share2/BLR.ovpa.11.12.i/linx64/hpsw-oa/AgentFramework/cpp/src/Scheduler/ScheduleWorker.cpp:32
#16 0x00007fc17c04a418 in OvXplThread::ThreadInfo_t::Run() () from
/opt/OV/lib64/libOvXpl.so
#17 0x00007fc17c04a6a5 in runThread () from /opt/OV/lib64/libOvXpl.so
#18 0x00007fc17b22f7f1 in start_thread () from /lib64/libpthread.so.0



[root at martellvm6 ~]# virsh version
Compiled against library: libvir 0.9.4
Using library: libvir 0.9.4
Using API: QEMU 0.9.4
error: failed to get the hypervisor version
error: internal error Cannot find suitable emulator for x86_64

[root at martellvm6 ~]# cat /etc/issue
CentOS release 6.2 (Final)
Kernel \r on an \m

On 27.05.2013 08:20, Parakkal, Navin S wrote:
> Hi,
>   We are hitting a deadlock on libvirt on 0.9.4 version. Can someone help
us ?
> 
> 1) Is there any way to specify timeout for all the calls made by libvirt
especially virConnectOpenReadOnly
> 
> 2) Does virCommandRunAsync always fork a new process ?
> 
> 
> gdb) bt
> #0  0x00007fc17b53ce6e in __lll_lock_wait_private () from /lib64/libc.so.6
> #1  0x00007fc17b4e3f2d in _L_lock_2164 () from /lib64/libc.so.6
> #2  0x00007fc17b4e3ce7 in __tz_convert () from /lib64/libc.so.6
> #3  0x00007fc16e4bfdab in virLogMessage () from /usr/lib64/libvirt.so.0
> #4  0x00007fc16e4b092e in ?? () from /usr/lib64/libvirt.so.0
> #5  0x00007fc16e4b2836 in ?? () from /usr/lib64/libvirt.so.0
> #6  0x00007fc16e4b318c in virCommandRunAsync () from
/usr/lib64/libvirt.so.0
> #7  0x00007fc16e570705 in ?? () from /usr/lib64/libvirt.so.0
> #8  0x00007fc16e56a1f8 in ?? () from /usr/lib64/libvirt.so.0
> #9  0x00007fc16e55c3d2 in ?? () from /usr/lib64/libvirt.so.0
> #10 0x00007fc16e55db10 in ?? () from /usr/lib64/libvirt.so.0
> #11 0x00007fc16e52ed95 in ?? () from /usr/lib64/libvirt.so.0
> #12 0x00007fc16e52f957 in virConnectOpenReadOnly () from
/usr/lib64/libvirt.so.0
> #13 0x00007fc153cdb858 in
OalibvirtCollectionManager::libvirtColJobImpl::Run (this=0x7fc13182a610)
>     at ../libvirtColJobImpl.cpp:348
> #14 0x00007fc153cd6e83 in OalibvirtCollectionManager::libvirtColJob::Run
(this=<value optimized out>)
>     at ../libvirtColJob.cpp:36
> #15 0x00007fc17cc8ee36 in OAScheduler::ScheduleWorker::Run
(this=0x7fc15dd9ede0,
>     myThread=<value optimized out>)
>     at
/svn-share2/BLR.ovpa.11.12.i/linx64/hpsw-oa/AgentFramework/cpp/src/Scheduler/ScheduleWorker.cpp:32
> #16 0x00007fc17c04a418 in OvXplThread::ThreadInfo_t::Run() () from
/opt/OV/lib64/libOvXpl.so
> #17 0x00007fc17c04a6a5 in runThread () from /opt/OV/lib64/libOvXpl.so
> #18 0x00007fc17b22f7f1 in start_thread () from /lib64/libpthread.so.0
> 
> 
> 
> [root at martellvm6 ~]# virsh version
> Compiled against library: libvir 0.9.4
> Using library: libvir 0.9.4
> Using API: QEMU 0.9.4
> error: failed to get the hypervisor version
> error: internal error Cannot find suitable emulator for x86_64
> 
> [root at martellvm6 ~]# cat /etc/issue
> CentOS release 6.2 (Final)
> Kernel \r on an \m
I think this issue had been fixed a while (since 0.9.13):

commit d581313acf863b5f8c39ee9940ada83d680570b2
Author:     Jiri Denemark <jdenemar at redhat.com>
AuthorDate: Thu Jun 7 15:16:50 2012 +0200
Commit:     Jiri Denemark <jdenemar at redhat.com>
CommitDate: Fri Jun 8 10:09:54 2012 +0200

    util: Fix deadlock in virLogReset
    
    When libvirtd forks off a new child, the child then calls virLogReset(),
    which ends up closing file descriptors used as log outputs. However, we
    recently started logging closed file descriptors, which means we need to
    lock logging mutex which was already locked by virLogReset(). We don't
    really want to log anything when we are in the process of closing log
    outputs.

On 05/27/2013 12:50 AM, Michal Privoznik wrote:
> I think this issue had been fixed a while (since 0.9.13):
> 
> commit d581313acf863b5f8c39ee9940ada83d680570b2
> Author:     Jiri Denemark <jdenemar@redhat.com>
> AuthorDate: Thu Jun 7 15:16:50 2012 +0200
> Commit:     Jiri Denemark <jdenemar@redhat.com>
> CommitDate: Fri Jun 8 10:09:54 2012 +0200
> 
>     util: Fix deadlock in virLogReset
That deadlock has been fixed, but there is another one still present in
libvirt.git (I'm waiting one last copyright release in gnulib before I
can apply the fix to libvirt):
https://www.redhat.com/archives/libvir-list/2013-May/msg01600.html

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org