Hello, I configured libvirtd with the sanlock lock manager plugin: # rpm -qa | egrep "libvirt-0|sanlock-[01]" libvirt-lock-sanlock-0.9.4-23.el6_2.4.x86_64 sanlock-1.8-2.el6.x86_64 libvirt-0.9.4-23.el6_2.4.x86_64 # egrep -v "^#|^$" /etc/libvirt/qemu-sanlock.conf auto_disk_leases = 1 disk_lease_dir = "/var/lib/libvirt/sanlock" host_id = 4 # mount | grep sanlock /dev/mapper/kvm--shared-sanlock on /var/lib/libvirt/sanlock type gfs2 (rw,noatime,hostdata=jid=0) # cat /etc/sysconfig/sanlock SANLOCKOPTS="-R 1 -o 30" I increased the sanlock io_timeout to 30 seconds (default = 10), because the sanlock dir is on a GFS2 volume and can be blocked for some time while fencing and journal recovery takes place. With the default sanlock io timeout, I get lease timeouts because IO is blocked: Mar 5 15:37:14 raiti sanlock[5858]: 3318 s1 check_our_lease warning 79 last_success 3239 Mar 5 15:37:15 raiti sanlock[5858]: 3319 s1 check_our_lease failed 80 So far, all fine, but when I restart sanlock and libvirtd, it takes about 2 * 30 seconds = 1 minute before libvirtd is usable. "virsh list" hangs during this time. I can still live with that... But it gets worse after a reboot, when running a "virsh list" even takes a couple of minutes (like about 5 minutes) before it responds. After this initial time, virsh is responding normally, so it looks like an initialization issue to me. Is this a configuration issue, a bug, or expected behavior? Thanks for any advice... Frido -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20120313/e68f415d/attachment.htm>
On Tue, Mar 13, 2012 at 03:42:36PM +0100, Frido Roose wrote:> Hello, > > I configured libvirtd with the sanlock lock manager plugin: > > # rpm -qa | egrep "libvirt-0|sanlock-[01]" > libvirt-lock-sanlock-0.9.4-23.el6_2.4.x86_64 > sanlock-1.8-2.el6.x86_64 > libvirt-0.9.4-23.el6_2.4.x86_64 > > # egrep -v "^#|^$" /etc/libvirt/qemu-sanlock.conf > auto_disk_leases = 1 > disk_lease_dir = "/var/lib/libvirt/sanlock" > host_id = 4 > > # mount | grep sanlock > /dev/mapper/kvm--shared-sanlock on /var/lib/libvirt/sanlock type gfs2 > (rw,noatime,hostdata=jid=0) > > # cat /etc/sysconfig/sanlock > SANLOCKOPTS="-R 1 -o 30" > > I increased the sanlock io_timeout to 30 seconds (default = 10), because > the sanlock dir is on a GFS2 volume and can be blocked for some time while > fencing and journal recovery takes place. > With the default sanlock io timeout, I get lease timeouts because IO is > blocked: > Mar 5 15:37:14 raiti sanlock[5858]: 3318 s1 check_our_lease warning 79 > last_success 3239 > Mar 5 15:37:15 raiti sanlock[5858]: 3319 s1 check_our_lease failed 80 > > So far, all fine, but when I restart sanlock and libvirtd, it takes about 2 > * 30 seconds = 1 minute before libvirtd is usable. "virsh list" hangs > during this time. I can still live with that... > But it gets worse after a reboot, when running a "virsh list" even takes a > couple of minutes (like about 5 minutes) before it responds. After this > initial time, virsh is responding normally, so it looks like an > initialization issue to me. > > Is this a configuration issue, a bug, or expected behavior?Each libvirtd instance has a lease that it owns. When restarting libvirtd it tries to acquire this lease. I don't really understand why, but sanlock sometimes has to wait a very long time between starting & completing its lease acquisition. You'll probably have to ask the sanlock developers for an explanation of why Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 03/13/2012 10:42 PM, Frido Roose wrote:> Hello, > > I configured libvirtd with the sanlock lock manager plugin: > > # rpm -qa | egrep "libvirt-0|sanlock-[01]" > libvirt-lock-sanlock-0.9.4-23.el6_2.4.x86_64 > sanlock-1.8-2.el6.x86_64 > libvirt-0.9.4-23.el6_2.4.x86_64 > > # egrep -v "^#|^$" /etc/libvirt/qemu-sanlock.conf > auto_disk_leases = 1 > disk_lease_dir = "/var/lib/libvirt/sanlock" > host_id = 4 > > # mount | grep sanlock > /dev/mapper/kvm--shared-sanlock on /var/lib/libvirt/sanlock type gfs2 > (rw,noatime,hostdata=jid=0) > > # cat /etc/sysconfig/sanlock > SANLOCKOPTS="-R 1 -o 30" > > I increased the sanlock io_timeout to 30 seconds (default = 10), > because the sanlock dir is on a GFS2 volume and can be blocked for > some time while fencing and journal recovery takes place. > With the default sanlock io timeout, I get lease timeouts because IO > is blocked: > Mar 5 15:37:14 raiti sanlock[5858]: 3318 s1 check_our_lease > warning 79 last_success 3239 > Mar 5 15:37:15 raiti sanlock[5858]: 3319 s1 check_our_lease failed 80 > > So far, all fine, but when I restart sanlock and libvirtd, it takes > about 2 * 30 seconds = 1 minute before libvirtd is usable. "virsh > list" hangs during this time. I can still live with that... > But it gets worse after a reboot, when running a "virsh list" even > takes a couple of minutes (like about 5 minutes) before it responds. > After this initial time, virsh is responding normally, so it looks > like an initialization issue to me. > > Is this a configuration issue, a bug, or expected behavior?Hi Frido, I'm not sure whether you met a sanlock AVC error in your /var/log/audit/audit.log, could you check it and provide your selinux-policy version? in addition, you should turn on selinux bool value for sanlock, for example, # getsebool -a|grep sanlock virt_use_sanlock --> off # setsebool -P virt_use_sanlock on # getsebool -a|grep sanlock virt_use_sanlock --> on In addition, could you provide libvirt log as a attachment? please refer the following configuration: 1. /etc/libvirt/libvirtd.conf log_filters="1:libvirt 1:conf 1:locking" log_outputs="1:file:/var/log/libvirt/libvirtd.log" 2. service libvirtd restart 3. repeat your test steps Good Luck! Alex> > Thanks for any advice... > Frido > > > _______________________________________________ > libvirt-users mailing list > libvirt-users at redhat.com > https://www.redhat.com/mailman/listinfo/libvirt-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20120314/bac19816/attachment.htm>