Predrag Ivanovic
2016-Mar-04 15:24 UTC
[libvirt-users] [libvirt-1.3.2]'Disconnected from qemu:///session due to I/O error'
Hi.
Since the update to libvirt-1.3.2, virsh keeps disconnecting from
qemu:///session.
Weirdly enough, qemu:///session works fine for root, and qemu:///system works
for both the user and the root.
With the same configuration(tarball with files from /etc/libvirt attached),
libvirt-1.3.1 works without problems.
From the virsh debug log(attached):
---
grep error virsh.log.1.3.2
2016-03-03 19:19:20.338+0000: 5589: error : virNetSocketReadWire:1613 : End of
file while reading data: Input/output error
2016-03-03 19:19:46.261+0000: 5578: error :
virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A
different callback was requested
2016-03-03 19:22:59.308+0000: 14487: error : virNetSocketReadWire:1613 : End of
file while reading data: Input/output error
2016-03-03 19:23:29.247+0000: 14486: error :
virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A
different callback was requested
2016-03-03 19:24:49.521+0000: 14487: error : virKeepAliveTimerInternal:143 :
internal error: connection closed due to keepalive timeout
2016-03-03 19:25:17.043+0000: 14486: error :
virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A
different callback was requested
2016-03-03 19:25:47.110+0000: 14487: error : virNetSocketReadWire:1613 : End of
file while reading data: Input/output error
2016-03-03 19:26:18.573+0000: 14486: error :
virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A
different callback was requested
2016-03-03 19:28:24.142+0000: 14665: error : virNetSocketReadWire:1613 : End of
file while reading data: Input/output error
2016-03-03 19:28:55.091+0000: 14664: error :
virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A
different callback was requested
---
From the virt-manager.log (attached):
---
[Thu, 03 Mar 2016 18:39:38 virt-manager 24388] DEBUG (connection:1310) Error
polling connection qemu:///session
if ret is None: raise libvirtError ('virNodeGetInfo() failed',
conn=self)
libvirtError: internal error: client socket is closed
if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny()
failed', conn=self)
libvirtError: internal error: client socket is closed
[Thu, 03 Mar 2016 19:07:14 virt-manager 5191] ERROR (create:667) Error setting
create wizard conn state.
if ret is None: raise libvirtError ('virConnectGetCapabilities()
failed', conn=self)
libvirtError: internal error: client socket is closed
[Thu, 03 Mar 2016 19:07:15 virt-manager 5191] DEBUG (connection:1310) Error
polling connection qemu:///session
if ret is None: raise libvirtError ('virNodeGetInfo() failed',
conn=self)
libvirtError: internal error: client socket is closed
if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny()
failed', conn=self)
libvirtError: internal error: client socket is closed
[Thu, 03 Mar 2016 19:08:19 virt-manager 5191] DEBUG (connection:1310) Error
polling connection qemu:///session
if ret is None: raise libvirtError ('virNodeGetInfo() failed',
conn=self)
libvirtError: internal error: client socket is closed
if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny()
failed', conn=self)
libvirtError: internal error: client socket is closed
[Thu, 03 Mar 2016 19:11:19 virt-manager 5191] DEBUG (connection:1310) Error
polling connection qemu:///system
if ret is None: raise libvirtError ('virNodeGetInfo() failed',
conn=self)
libvirtError: internal error: client socket is closed
if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny()
failed', conn=self)
libvirtError: internal error: client socket is closed
---
qemu:///system errors above are due to restarting libvirtd, once virt-manager
connected to it, it stayed connected.
This is with qemu-2.5.0.
I could try libvirt master from git, but will it break
libvirt-python/virt-manager?
Btw, is building libvirt with '--enable-debug' sufficient for useful gdb
debug report, or do you need anything else?
Thank you.
Pedja
Michal Privoznik
2016-Mar-05 10:16 UTC
Re: [libvirt-users] [libvirt-1.3.2]'Disconnected from qemu:///session due to I/O error'
On 04.03.2016 16:24, Predrag Ivanovic wrote:> Hi. > > Since the update to libvirt-1.3.2, virsh keeps disconnecting from qemu:///session. > Weirdly enough, qemu:///session works fine for root, and qemu:///system works for both the user and the root. > With the same configuration(tarball with files from /etc/libvirt attached), libvirt-1.3.1 works without problems. > > From the virsh debug log(attached): > --- > grep error virsh.log.1.3.2 > 2016-03-03 19:19:20.338+0000: 5589: error : virNetSocketReadWire:1613 : End of file while reading data: Input/output error > 2016-03-03 19:19:46.261+0000: 5578: error : virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A different callback was requested > 2016-03-03 19:22:59.308+0000: 14487: error : virNetSocketReadWire:1613 : End of file while reading data: Input/output error > 2016-03-03 19:23:29.247+0000: 14486: error : virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A different callback was requested > 2016-03-03 19:24:49.521+0000: 14487: error : virKeepAliveTimerInternal:143 : internal error: connection closed due to keepalive timeout > 2016-03-03 19:25:17.043+0000: 14486: error : virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A different callback was requested > 2016-03-03 19:25:47.110+0000: 14487: error : virNetSocketReadWire:1613 : End of file while reading data: Input/output error > 2016-03-03 19:26:18.573+0000: 14486: error : virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A different callback was requested > 2016-03-03 19:28:24.142+0000: 14665: error : virNetSocketReadWire:1613 : End of file while reading data: Input/output error > 2016-03-03 19:28:55.091+0000: 14664: error : virConnectUnregisterCloseCallback:1282 : Requested operation is not valid: A different callback was requestedThis looks like a dying session daemon to me. You can check if my suspicion is right by running the session daemon by hand. From a console run: $ /usr/sbin/libvirtd Usually, connecting to qemu:///session does the same, except it appends '--timeout=30' to tell the session daemon to die after 30 seconds of inactivity. And here's the problem - there should be a keepalive running preventing daemon from dying.> --- > From the virt-manager.log (attached): > --- > [Thu, 03 Mar 2016 18:39:38 virt-manager 24388] DEBUG (connection:1310) Error polling connection qemu:///session > if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self) > libvirtError: internal error: client socket is closed > if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny() failed', conn=self) > libvirtError: internal error: client socket is closed > [Thu, 03 Mar 2016 19:07:14 virt-manager 5191] ERROR (create:667) Error setting create wizard conn state. > if ret is None: raise libvirtError ('virConnectGetCapabilities() failed', conn=self) > libvirtError: internal error: client socket is closed > [Thu, 03 Mar 2016 19:07:15 virt-manager 5191] DEBUG (connection:1310) Error polling connection qemu:///session > if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self) > libvirtError: internal error: client socket is closed > if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny() failed', conn=self) > libvirtError: internal error: client socket is closed > [Thu, 03 Mar 2016 19:08:19 virt-manager 5191] DEBUG (connection:1310) Error polling connection qemu:///session > if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self) > libvirtError: internal error: client socket is closed > if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny() failed', conn=self) > libvirtError: internal error: client socket is closed > [Thu, 03 Mar 2016 19:11:19 virt-manager 5191] DEBUG (connection:1310) Error polling connection qemu:///system > if ret is None: raise libvirtError ('virNodeGetInfo() failed', conn=self) > libvirtError: internal error: client socket is closed > if ret == -1: raise libvirtError ('virConnectDomainEventDeregisterAny() failed', conn=self) > libvirtError: internal error: client socket is closed > --- > qemu:///system errors above are due to restarting libvirtd, once virt-manager connected to it, it stayed connected. > > This is with qemu-2.5.0. > > I could try libvirt master from git, but will it break libvirt-python/virt-manager?No. The APIs are backward compatible.> Btw, is building libvirt with '--enable-debug' sufficient for useful gdb debug report, or do you need anything else?The best would be to disable any optimization and include gdb debuging info: libvirt.git $ export CFLAGS="-O0 -ggdb3" ; ./configure --enable-debug && make Michal
Predrag Ivanovic
2016-Mar-05 18:02 UTC
Re: [libvirt-users] [libvirt-1.3.2]'Disconnected from qemu:///session due to I/O error'
On Sat, 05 Mar 2016 11:16:21 +0100 Michal Privoznik wrote:>>This looks like a dying session daemon to me. You can check if my >suspicion is right by running the session daemon by hand. From a console >run: > >$ /usr/sbin/libvirtd > >Usually, connecting to qemu:///session does the same, except it appends >'--timeout=30' to tell the session daemon to die after 30 seconds of >inactivity. And here's the problem - there should be a keepalive running >preventing daemon from dying. >libvirt-1.3.1 root@deus:~/bin#> ps aux | grep libvirtd pedja 3122 0.0 0.0 1027672 2376 ? Sl Mar03 0:03 /usr/sbin/libvirtd --timeout=30 root 19814 0.0 0.1 1298708 5500 ? Sl Mar03 0:00 /usr/sbin/libvirtd -d -v -f /etc/libvirt/libvirtd.conf -p /var/run/libvirt/libvirtd.pid ------- libvirt-git-92ec2e5 root@deus:~/bin#> ps aux | grep libvirtd root 13553 0.0 0.4 972948 16552 ? Sl 16:02 0:00 /usr/sbin/libvirtd -d -v -f /etc/libvirt/libvirtd.conf -p /var/run/libvirt/libvirtd.pid pedja 13723 0.0 0.4 965248 18508 ? Sl 16:03 0:00 /usr/sbin/libvirtd --timeout=30 ------- libvirt-1.3.2 root@deus:~/bin#> ps aux | grep libvirtd root 16778 0.3 0.3 383212 15700 ? Sl 17:10 0:00 /usr/sbin/libvirtd -d -v -f /etc/libvirt/libvirtd.conf -p /var/run/libvirt/libvirtd.pid pedja 16975 3.2 0.4 637588 17964 ? Sl 17:11 0:00 /usr/sbin/libvirtd --timeout=30 Only the session daemon from libvirt-1.3.2 dies randomly, 1.3.1 and libvirt-git-92ec2e5 are stable (no disconnects). I managed to hook gdb to session daemon above after a few tries, and gdb log ('thread apply all bt') and virsh.log are attached. I'll be running libvirt from git master for now, but I have both it and 1.3.2 built with debug symbols, if you need more information. Pedja