Zhan Adlun
2019-Mar-03 03:23 UTC
[libvirt-users] virsh hangs when backup jobs run on the kvm host
Hi,
Has anyone met the same problem that virsh hangs when there are some backup
jobs running on the kvm host? I use dirty bitmap supported by qemu to do backups
every 6 minitues. About a few hours later, I ran 'virsh list', It hung.
Then I stopped the backup jobs ?nothing was getting better. I had checked the
libvirt logs ?but found nothing useful .
Maybe the high I/O caused the problem? or there are connection number limits
when using virsh, because I use virsh to check the backup jobs are still running
every 10 seconds?
My os version is ' Ubuntu 12.04 LTS Release: 12.04 Codename:
precise,Linux version 4.1.0-generic (gcc version 4.6.3 (Ubuntu/Linaro
4.6.3-1ubuntu5) ) #1 SMP Wed Nov 9 02:04:23 CST 2016'. KVM version is
'QEMU emulator version 2.7.1(Debian 1:2.7+dfsg-3)'. Hoping for you
reply. Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20190303/0e7d0eee/attachment.htm>
Michal Prívozník
2019-Mar-06 15:26 UTC
[libvirt-users] virsh hangs when backup jobs run on the kvm host
On 3/3/19 4:23 AM, Zhan Adlun wrote:> Hi, > Has anyone met the same problem that virsh hangs when there are some backup jobs running on the kvm host? I use dirty bitmap supported by qemu to do backups every 6 minitues. About a few hours later, I ran 'virsh list', It hung. Then I stopped the backup jobs ?nothing was getting better. I had checked the libvirt logs ?but found nothing useful . > Maybe the high I/O caused the problem? or there are connection number limits when using virsh, because I use virsh to check the backup jobs are still running every 10 seconds? > My os version is ' Ubuntu 12.04 LTS Release: 12.04 Codename: precise,Linux version 4.1.0-generic (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #1 SMP Wed Nov 9 02:04:23 CST 2016'. KVM version is 'QEMU emulator version 2.7.1(Debian 1:2.7+dfsg-3)'. Hoping for you reply. Thanks!This sounds like a deadlock. If you can attach gdb to libvirtd once virsh hangs and run 't a a bt' and share the output, that'd be great. Michal
Eric Blake
2019-Mar-06 15:43 UTC
[libvirt-users] virsh hangs when backup jobs run on the kvm host
On 3/2/19 9:23 PM, Zhan Adlun wrote:> Hi, > Has anyone met the same problem that virsh hangs when there are some backup jobs running on the kvm host? I use dirty bitmap supported by qemu to do backups every 6 minitues. About a few hours later, I ran 'virsh list', It hung. Then I stopped the backup jobs ?nothing was getting better. I had checked the libvirt logs ?but found nothing useful .How are you using dirty bitmaps? As my new virDomainBackupBegin() API has not landed yet, I suspect you are using virQemuMonitorCommand() (virsh qemu-monitor-command) backdoors, which is explicitly unsupported (that is, if it breaks, you get to debug it - it exists for development work, and while 'query-*' QMP commands should be safe, commands that alter qemu state such as kicking off a backup job may break libvirt expectations), and which is WHY I'm trying to get incremental backup API officially supported. A precise sequence of commands that you issued in order to reproduce the virsh hang may be useful in debugging this.> Maybe the high I/O caused the problem? or there are connection number limits when using virsh, because I use virsh to check the backup jobs are still running every 10 seconds? > My os version is ' Ubuntu 12.04 LTS Release: 12.04 Codename: precise,Linux version 4.1.0-generic (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #1 SMP Wed Nov 9 02:04:23 CST 2016'. KVM version is 'QEMU emulator version 2.7.1(Debian 1:2.7+dfsg-3)'. Hoping for you reply. Thanks!Hmm, my incremental backup API will require qemu 4.0; there have been a number of fixes in the qemu bitmap tracking that have landed since qemu 2.7.1. It could also be that you've hit an actual qemu bug where the QMP monitor has gone unresponsive. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20190306/76bdc143/attachment.sig>
Zhan Adlun
2019-Mar-07 02:17 UTC
[libvirt-users] 回复: Re: virsh hangs when backup jobs run on the kvm host
Actually,I use the backdoors as you have mentioned. I use 'virsh
qemu-monitor-command' to submit backup jobs and use 'virsh
qemu-monitor-event DOMAIN --timestamp --loop' to monitor the jobs through
the QMP protocol.
for example :
to do a full backup
virsh qemu-monitor-command DOMAIN --pretty { "execute":
"transaction",
"arguments": {
"actions": [
{"type": "block-dirty-bitmap-add",
"data": {"node": "drive0",
"name": "bitmap0"} },
{"type": "drive-backup",
"data": {"device": "drive0",
"target": "/path/to/full_backup.img",
"sync": "full", "format":
"qcow2"} }
]
}
}
refer to https://wiki.qemu.org/Features/IncrementalBackup
Unfortunately, the minitor process 'virsh qemu-monitor-event DOMAIN
--timestamp --loop' was failed to be killed. A few hours later ,there were
hundreds of monitor processes running(about 200-300), whice caused the
'virsh list' hung. After I have killed the processes, 'virsh
list' works well.
________________________________
zhanhouliang@outlook.com
发件人： Eric Blake<mailto:eblake@redhat.com>
发送时间： 2019-03-06 23:43
收件人： Zhan Adlun<mailto:zhanhouliang@outlook.com>;
libvirt-users@redhat.com<mailto:libvirt-users@redhat.com>
主题： Re: [libvirt-users] virsh hangs when backup jobs run on the kvm host
On 3/2/19 9:23 PM, Zhan Adlun wrote:> Hi,
> Has anyone met the same problem that virsh hangs when there are some
backup jobs running on the kvm host? I use dirty bitmap supported by qemu to do
backups every 6 minitues. About a few hours later, I ran 'virsh list',
It hung. Then I stopped the backup jobs ，nothing was getting better. I had
checked the libvirt logs ，but found nothing useful .
How are you using dirty bitmaps? As my new virDomainBackupBegin() API
has not landed yet, I suspect you are using virQemuMonitorCommand()
(virsh qemu-monitor-command) backdoors, which is explicitly unsupported
(that is, if it breaks, you get to debug it - it exists for development
work, and while 'query-*' QMP commands should be safe, commands that
alter qemu state such as kicking off a backup job may break libvirt
expectations), and which is WHY I'm trying to get incremental backup API
officially supported.
A precise sequence of commands that you issued in order to reproduce the
virsh hang may be useful in debugging this.
> Maybe the high I/O caused the problem? or there are connection number
limits when using virsh, because I use virsh to check the backup jobs are still
running every 10 seconds?
> My os version is ' Ubuntu 12.04 LTS Release: 12.04 Codename:
precise,Linux version 4.1.0-generic (gcc version 4.6.3 (Ubuntu/Linaro
4.6.3-1ubuntu5) ) #1 SMP Wed Nov 9 02:04:23 CST 2016'. KVM version is
'QEMU emulator version 2.7.1(Debian 1:2.7+dfsg-3)'. Hoping for you
reply. Thanks!
Hmm, my incremental backup API will require qemu 4.0; there have been a
number of fixes in the qemu bitmap tracking that have landed since qemu
2.7.1. It could also be that you've hit an actual qemu bug where the QMP
monitor has gone unresponsive.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org