[moderator note: .pngs were stripped to avoid overwhelming the mail
server and list recipients with 1M of data]
-------- Forwarded Message --------
Date: Wed, 30 Mar 2016 11:25:09 +0200
Message-ID:
<20160330112509.Horde.0Oad-ZfkzdZnouXf-VkEmw1@webmailperso.univ-brest.fr>
From: villeneu@kassis.univ-brest.fr
To: Franky Van Liedekerke <liedekef@telenet.be>
Cc: libvirt-users@redhat.com
Subject: Re: [libvirt-users] VM crash and lock manager
References:
<20160329192740.Horde.BZXrmDPzMBshASPvOs_x4Q6@webmailperso.univ-brest.fr>
<20160329231311.1ae867a1@telenet.be>
In-Reply-To: <20160329231311.1ae867a1@telenet.be>
---
Franky Van Liedekerke <liedekef@telenet.be> a écrit :
> On Tue, 29 Mar 2016 19:27:40 +0200
> villeneu@kassis.univ-brest.fr wrote:
>
>> Hello
>>
>> I changed my hypervisors ( 8) from centos 6.0 to fedora 23 and I
>> added locks to prevent starting VM on multiple nodes, and it's work
>> well.
>>
>> Since I changed the OS some old VMs ( centos 5.x and 6.x ) with very
>> old kernel seem very instable and crash very often ( after one or two
>> days ). Before they never have these problems.
>>
>> I saw, perhaps the reason is the dev is too slow ??? see
screenshot>> joined
>>
>> A major problem with these crashs is that the VMs couldn't be
>> destroyed by the virsh command, the qemu process is notified as
>> defunct by the ps command.
>>
>> with virsh destroy VM
>> I often get
>>
>> Failed to terminate process xxx with SIGTERM: Device or resource busy
>> and the VM is still in the list
>>
>> If I try to remove all the file associated with the VM in /var/run or
>> /var/lib....channel .. It doesn't give results The
>> VM is still running as defunct and after a libvirt restart , libvirt
>> and virsh command are pending.
>>
>> With sanlock it's a problem because the is never release and
it's
>> impossible to restart the VM. I can't remove the VM in the sanlock
>> list. I tried kill commands, rm ... but without a real success.
>>
>> I tried many command with sanlock perhaps I misunderstood some
>> commands but I never succeed in removing lock on the VMs.
>>
>> sanlock client rem_lockspace -r
>>
__LIBVIRT__DISKS__:b90b9c61e2d6413077205907ffb3281a:/var/lib/libvirt/images/POOL_ADMIN/sanlock/b90b9c61e2d6413077205907ffb3281a:0:4>> -p 9383
>>
>> So I'm trying to use lock_manager to replace the sanlock but
>> lock_manager doesn't have tools to retrieve the lock / VM name in
>> case of failure or crash.
>>
>> My questions are, is there a way to release or to force to release a
>> lock with sanlock even if the qemu process is defunc and a way to
>> restart libvirt in normal usage ?
>>
>> Perhaps by the API it`s possible to forece the release the lock ?
>>
>> Is there tools to retreive or calculate SHA256 lock for a VM with the
>> lock_manager lock. I saw it's the lock is a SHA256 hash of the path
>> and the VM name but I nerver get the good hash when I try by the hand.
>>
>> Is there tools to retreive or calculate SHA256 the lock for a VM with
>> the lock_manager lock. I saw that the lock is a SHA256 hash of the
>> path and the VM name but I never get the good hash when I try by the
>> hand.
>>
>> Thanks.
>> Sorry for my poor english ... I 'am french
>> Michel
>
> It seems the lock held by sanlock isn't being released, so the vm
can't
> be shut down properly. Maybe some selinux issue?
> I've been using virtlockd without troubles for more than a year already
> (without selinux). For the locks, see if the lslocks command helps you.
>
> Franky
>
> _______________________________________________
> libvirt-users mailing list
>
libvirt-users@redhat.comhttps://www.redhat.com/mailman/listinfo/libvirt-users
I do not use selinux too, I preffered sanlock at the beginning because
sanlock comes with tools to remove locks. But when VMs crash abnormaly or
freeze ( see the sreenshot I missed to attach it yesterday ) there is
no way to unlease or remove the locks, they are presents even if I kill
the process qemu and I can't restart the VM because the lock is still here.
So I try now lock-manager, but do you know exactly how the lock is
calculate . In case of problem I want to be able to retrieve the lock
coupled with the VM.
I have seen that it is a hash SHA256 of the path and the name, but if a try
to do by the hand a sha256 of the complete path of the VM I don't have
the correct value ( the same value as lock_manager ).
Any Ideas ?
Thanks
crash1.png
crash2.png
