libvirt users - Apr 2013 - Libvirtd dead, pid still exists. ( Problem might be with TLS interface of libvirtd )

Dear All,

Please find few minutes from your time and guide us with some pointers if
possible.

We are facing a libvirtd crash when we are trying to connect to qemu by
default TLS transport.

# virsh -c qemu+tls://localhost/system version
error: authentication failed: TLS handshake failed A TLS packet with
unexpected length was received.
error: failed to connect to the hypervisor

I used my own CA and certificates (generated using
http://libvirt.org/remote.html#Remote_libvirtd_configuration on Redhat PC)
on both Kontron PC and our Board (with patched libvirt). Libvirtd.conf was
modified so that libvirt is listening all IPs using default IP (so that it
was possible to use same certificates on all machines)

These directories and files created and used.
/etc/pki/CA/cacert.pem
/etc/pki/libvirt/private/serverkey.pem
/etc/pki/libvirt/servercert.pem
/etc/pki/libvirt/private/clientkey.pem
/etc/pki/libvirt/clientcert.pem

TLS connection worked fine with Kontron PC

# virsh -c qemu+tls://localhost/system version
Compiled against library: libvir 0.9.5
Using library: libvir 0.9.5
Using API: QEMU 0.9.5
Running hypervisor: QEMU 0.12.1

But libvirt crashed on our Board (using libvirt 0.10.2,
gnutls-2.10.5-1_WR4.3.x86_64 and libudev-161-4 rpms )

# virsh -c qemu+tls://localhost/system version
error: authentication failed: TLS handshake failed A TLS packet with
unexpected length was received.
error: failed to connect to the hypervisor

GDB:

Program received signal SIGABRT, Aborted.
0x00007f8591246005 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007f8591246005 in raise () from /lib64/libc.so.6
#1  0x00007f8591248e40 in abort () from /lib64/libc.so.6
#2  0x00007f8592a2fdc5 in ?? () from /lib64/libgcrypt.so.11
#3  0x00007f8592a303d5 in ?? () from /lib64/libgcrypt.so.11
#4  0x00007f8592a35697 in ?? () from /lib64/libgcrypt.so.11
#5  0x00007f8592a3579c in ?? () from /lib64/libgcrypt.so.11
#6  0x00007f8592a30a65 in ?? () from /lib64/libgcrypt.so.11
#7  0x00007f8592a30aa9 in ?? () from /lib64/libgcrypt.so.11
#8  0x00007f8592a30b19 in ?? () from /lib64/libgcrypt.so.11
#9  0x00007f8592a735df in ?? () from /lib64/libgcrypt.so.11
#10 0x00007f8592a7365f in ?? () from /lib64/libgcrypt.so.11
#11 0x00007f8592a6025a in ?? () from /lib64/libgcrypt.so.11
#12 0x00007f8592a6045a in ?? () from /lib64/libgcrypt.so.11
#13 0x00007f8592a3c1ef in ?? () from /lib64/libgcrypt.so.11
#14 0x00007f8592cd9d8c in ?? () from /usr/lib64/libgnutls.so.26
#15 0x00007f8592cc5e7a in ?? () from /usr/lib64/libgnutls.so.26
#16 0x00007f8592ccddd6 in ?? () from /usr/lib64/libgnutls.so.26
#17 0x00007f8592cce67f in ?? () from /usr/lib64/libgnutls.so.26
#18 0x00007f8592ccedaf in ?? () from /usr/lib64/libgnutls.so.26
#19 0x00007f8592cbaf85 in ?? () from /usr/lib64/libgnutls.so.26
#20 0x00007f8592cb6c55 in ?? () from /usr/lib64/libgnutls.so.26
#21 0x00007f8592cb7437 in gnutls_handshake () from
/usr/lib64/libgnutls.so.26
#22 0x00007f8593a5961b in virNetTLSSessionHandshake () from
/usr/lib64/libvirt.so.0

Please let us know if it is a known problem. If not, I will raise a new bug
for the same. ( Atleast I coulnt find the the known issues )

Thanks and Regards,
Shree Duth Awasthi.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20130405/cb4175fa/attachment.htm>

On 05.04.2013 09:04, SHREE DUTH AWASTHI wrote:
> GDB:
>  
> Program received signal SIGABRT, Aborted.
> 0x00007f8591246005 in raise () from /lib64/libc.so.6
> (gdb) bt
> #0  0x00007f8591246005 in raise () from /lib64/libc.so.6
> #1  0x00007f8591248e40 in abort () from /lib64/libc.so.6
> #2  0x00007f8592a2fdc5 in ?? () from /lib64/libgcrypt.so.11
> #3  0x00007f8592a303d5 in ?? () from /lib64/libgcrypt.so.11
> #4  0x00007f8592a35697 in ?? () from /lib64/libgcrypt.so.11
> #5  0x00007f8592a3579c in ?? () from /lib64/libgcrypt.so.11
> #6  0x00007f8592a30a65 in ?? () from /lib64/libgcrypt.so.11
> #7  0x00007f8592a30aa9 in ?? () from /lib64/libgcrypt.so.11
> #8  0x00007f8592a30b19 in ?? () from /lib64/libgcrypt.so.11
> #9  0x00007f8592a735df in ?? () from /lib64/libgcrypt.so.11
> #10 0x00007f8592a7365f in ?? () from /lib64/libgcrypt.so.11
> #11 0x00007f8592a6025a in ?? () from /lib64/libgcrypt.so.11
> #12 0x00007f8592a6045a in ?? () from /lib64/libgcrypt.so.11
> #13 0x00007f8592a3c1ef in ?? () from /lib64/libgcrypt.so.11
> #14 0x00007f8592cd9d8c in ?? () from /usr/lib64/libgnutls.so.26
> #15 0x00007f8592cc5e7a in ?? () from /usr/lib64/libgnutls.so.26
> #16 0x00007f8592ccddd6 in ?? () from /usr/lib64/libgnutls.so.26
> #17 0x00007f8592cce67f in ?? () from /usr/lib64/libgnutls.so.26
> #18 0x00007f8592ccedaf in ?? () from /usr/lib64/libgnutls.so.26
> #19 0x00007f8592cbaf85 in ?? () from /usr/lib64/libgnutls.so.26
> #20 0x00007f8592cb6c55 in ?? () from /usr/lib64/libgnutls.so.26
> #21 0x00007f8592cb7437 in gnutls_handshake () from
> /usr/lib64/libgnutls.so.26
> #22 0x00007f8593a5961b in virNetTLSSessionHandshake () from
> /usr/lib64/libvirt.so.0
>  
The backstrace shows problem lies in libgcrypt library. So unless
libvirt is overwriting some random memory areas, it's a libgcrypt's bug.

Michal