llilulu
2018-Mar-08 04:54 UTC
[libvirt-users] Statistics domain memory block when domain shutdown
Hi
My libvirt version is 3.4.0,host system is centos 7.4 ,kernel is
3.10.0-693.el7.x86_64 , when I shutdown domain in virtual system, My program
call virDomainMemoryStats, My program blocked in this api. the call stack is
#0 0x00007ff242d78a3d in poll () from /lib64/libc.so.6
#1 0x00007ff243755ce8 in virNetClientIOEventLoop () from /lib64/libvirt.so.0
#2 0x00007ff24375654b in virNetClientSendInternal () from /lib64/libvirt.so.0#3
0x00007ff2437579b3 in virNetClientSendWithReply () from /lib64/libvirt.so.0
#4 0x00007ff2437581c2 in virNetClientProgramCall () from /lib64/libvirt.so.0#5
0x00007ff24372d392 in callFull.isra.3 () from /lib64/libvirt.so.0
#6 0x00007ff24373b8a1 in remoteDomainMemoryStats () from /lib64/libvirt.so.0
#7 0x00007ff2436f14e2 in virDomainMemoryStats () from /lib64/libvirt.so.0
#8 0x00007ff22ee975c2 in VM::getMemoryStats (this=0x7ff1d4064ae0,
Begin I think is my program problem, But when I use virsh domstats cmd ,
The cmd block too. In my program, I set timeout event for virConnectPtr. But my
program blocked all the time. I find my system log, I find some libvirtd error:
libvirtd: 2018-03-08 04:25:13.144+0000: 1289: error :
virKeepAliveTimerInternal:143 : internal error: connection closed due to
keepalive timeout
Mar 7 23:33:05 localhost libvirtd: 2018-03-08 04:33:05.738+0000: 1289: error :
virNetSocketReadWire:1808 : End of file while reading data: Input/output error
Mar 7 23:33:14 localhost libvirtd: 2018-03-08 04:33:14.253+0000: 1289: error :
virNetSocketReadWire:1808 : End of file while reading data: Input/output error
Mar 7 23:33:16 localhost libvirtd: 2018-03-08 04:33:16.962+0000: 1289: error :
virNetSocketReadWire:1808 : End of file while reading data: Input/output error
Mar 7 23:33:25 localhost libvirtd: 2018-03-08 04:33:25.879+0000: 1300: warning
: qemuDomainObjBeginJobInternal:3847 : Cannot start job (query, none) for domain
d7bb3005-4fa4-4c3a-8802-ab55464d074b; current job is (query, none) owned by
(1299 remoteDispatchDomainMemoryStats, 0 <null>) for (3680s, 0s)
Mar 7 23:33:25 localhost libvirtd: 2018-03-08 04:33:25.879+0000: 1300: error :
qemuDomainObjBeginJobInternal:3859 : Timed out during operation: cannot acquire
state change lock (held by remoteDispatchDomainMemoryStats)
Mar 7 23:33:25 localhost libvirtd: 2018-03-08 04:33:25.879+0000: 1300: warning
: qemuGetProcessInfo:1428 : cannot parse process status data
Mar 7 23:33:25 localhost libvirtd: 2018-03-08 04:33:25.880+0000: 1300: error :
virNetDevTapInterfaceStats:751 : internal error: /proc/net/dev: Interface not
found
Mar 7 23:33:41 localhost libvirtd: 2018-03-08 04:33:41.826+0000: 1298: warning
: qemuDomainObjBeginJobInternal:3847 : Cannot start job (query, none) for domain
d7bb3005-4fa4-4c3a-8802-ab55464d074b; current job is (query, none) owned by
(1299 remoteDispatchDomainMemoryStats, 0 <null>) for (3696s, 0s)Mar 7
23:33:41 localhost libvirtd: 2018-03-08 04:33:41.826+0000: 1298: error :
qemuDomainObjBeginJobInternal:3859 : Timed out during operation: cannot acquire
state change lock (held by remoteDispatchDomainMemoryStats)
Is this problem a bug or I can't statistics some specifi status domain?
Thanks
Michal Privoznik
2018-Mar-09 12:02 UTC
Re: [libvirt-users] Statistics domain memory block when domain shutdown
On 03/08/2018 05:54 AM, llilulu wrote:> Hi > My libvirt version is 3.4.0,host system is centos 7.4 ,kernel is 3.10.0-693.el7.x86_64 , when I shutdown domain in virtual system, My program call virDomainMemoryStats, My program blocked in this api. the call stack is > > #0 0x00007ff242d78a3d in poll () from /lib64/libc.so.6 > #1 0x00007ff243755ce8 in virNetClientIOEventLoop () from /lib64/libvirt.so.0 > #2 0x00007ff24375654b in virNetClientSendInternal () from /lib64/libvirt.so.0#3 0x00007ff2437579b3 in virNetClientSendWithReply () from /lib64/libvirt.so.0 > #4 0x00007ff2437581c2 in virNetClientProgramCall () from /lib64/libvirt.so.0#5 0x00007ff24372d392 in callFull.isra.3 () from /lib64/libvirt.so.0 > #6 0x00007ff24373b8a1 in remoteDomainMemoryStats () from /lib64/libvirt.so.0 > #7 0x00007ff2436f14e2 in virDomainMemoryStats () from /lib64/libvirt.so.0 > #8 0x00007ff22ee975c2 in VM::getMemoryStats (this=0x7ff1d4064ae0,This is a client stack trace and it only shows that client called virDomainMemoryStats(). Can you get daemon stack trace?> > Begin I think is my program problem, But when I use virsh domstats cmd , The cmd block too. In my program, I set timeout event for virConnectPtr. But my program blocked all the time. I find my system log, I find some libvirtd error: > libvirtd: 2018-03-08 04:25:13.144+0000: 1289: error : virKeepAliveTimerInternal:143 : internal error: connection closed due to keepalive timeout > Mar 7 23:33:05 localhost libvirtd: 2018-03-08 04:33:05.738+0000: 1289: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error > Mar 7 23:33:14 localhost libvirtd: 2018-03-08 04:33:14.253+0000: 1289: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error > Mar 7 23:33:16 localhost libvirtd: 2018-03-08 04:33:16.962+0000: 1289: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error > Mar 7 23:33:25 localhost libvirtd: 2018-03-08 04:33:25.879+0000: 1300: warning : qemuDomainObjBeginJobInternal:3847 : Cannot start job (query, none) for domain d7bb3005-4fa4-4c3a-8802-ab55464d074b; current job is (query, none) owned by (1299 remoteDispatchDomainMemoryStats, 0 <null>) for (3680s, 0s) > Mar 7 23:33:25 localhost libvirtd: 2018-03-08 04:33:25.879+0000: 1300: error : qemuDomainObjBeginJobInternal:3859 : Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMemoryStats) > Mar 7 23:33:25 localhost libvirtd: 2018-03-08 04:33:25.879+0000: 1300: warning : qemuGetProcessInfo:1428 : cannot parse process status data > Mar 7 23:33:25 localhost libvirtd: 2018-03-08 04:33:25.880+0000: 1300: error : virNetDevTapInterfaceStats:751 : internal error: /proc/net/dev: Interface not found > Mar 7 23:33:41 localhost libvirtd: 2018-03-08 04:33:41.826+0000: 1298: warning : qemuDomainObjBeginJobInternal:3847 : Cannot start job (query, none) for domain d7bb3005-4fa4-4c3a-8802-ab55464d074b; current job is (query, none) owned by (1299 remoteDispatchDomainMemoryStats, 0 <null>) for (3696s, 0s)Mar 7 23:33:41 localhost libvirtd: 2018-03-08 04:33:41.826+0000: 1298: error : qemuDomainObjBeginJobInternal:3859 : Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMemoryStats) > > Is this problem a bug or I can't statistics some specifi status domain?This is very likely a bug that has been fixed. There were some fixes concerning statistics fetching. Please try to reproduce with the latest git HEAD. Michal