libvirt users - Feb 2014 - Failed to terminate process X with SIGKILL: Device or resource busy

Hi,

I'm using libvirt/kvm with openstack. When vm has CPU stall, the instance
can't be destroyed. I have asked the question on openstack user mailing
list and received no answers in weeks.

Here is one example:

    # virsh destroy instance-00000085
    error: Failed to destroy domain instance-00000085
    error: Failed to terminate process 61222 with SIGKILL: Device or
resource busy

    # lsof -p 61222
    # ps -f -p 61222
    UID        PID  PPID  C STIME TTY          TIME CMD
    107      61222     1 68 Jan24 ?        9-23:06:19 [qemu-system-x86]
<defunct>

The log in /var/log/libvirt/libvirtd.log:
    2014-02-08 03:21:31.597+0000: 2177: error : virProcessKillPainfully:315
: Failed to terminate process 61222 with SIGKILL: Device or resource busy

Restart libvirt-bin service did not solve the problem.

virsh shutdown/reboot just hang. Maybe that is because the CPU in vm is
stalled.

I wonder if there is some way to destroy (or rebuild) dead instance without
rebooting the hypervisor. Any suggestions is welcome.

The software I am running:
hypervisor os: ubuntu 12.04.3
hypervisor kernel: 3.8.0-35-generic
vm os: ubuntu 12.04
vm kernel: 3.2.0-56-virtual
openstack: havana
libvirt: 1.1.1-0ubuntu8~cloud2

Thanks,
Yuanle

FYI, I have found a way to terminate instance:
    virsh schedinfo <domain> --set vcpu_quota=20000
After this, shutdown, reboot and destroy all works. Of course, this
disables the vcpu throttling. It was set to 5000, which may cause the CPU
stall problem in guest.

Thanks,
Yuanle



On Sat, Feb 8, 2014 at 11:34 AM, sylecn <sylecn@gmail.com> wrote:
> Hi,
>
> I'm using libvirt/kvm with openstack. When vm has CPU stall, the
instance
> can't be destroyed. I have asked the question on openstack user mailing
> list and received no answers in weeks.
>
> Here is one example:
>
>     # virsh destroy instance-00000085
>     error: Failed to destroy domain instance-00000085
>     error: Failed to terminate process 61222 with SIGKILL: Device or
> resource busy
>
>     # lsof -p 61222
>     # ps -f -p 61222
>     UID        PID  PPID  C STIME TTY          TIME CMD
>     107      61222     1 68 Jan24 ?        9-23:06:19 [qemu-system-x86]
> <defunct>
>
> The log in /var/log/libvirt/libvirtd.log:
>     2014-02-08 03:21:31.597+0000: 2177: error :
> virProcessKillPainfully:315 : Failed to terminate process 61222 with
> SIGKILL: Device or resource busy
>
> Restart libvirt-bin service did not solve the problem.
>
> virsh shutdown/reboot just hang. Maybe that is because the CPU in vm is
> stalled.
>
> I wonder if there is some way to destroy (or rebuild) dead instance
> without rebooting the hypervisor. Any suggestions is welcome.
>
> The software I am running:
> hypervisor os: ubuntu 12.04.3
> hypervisor kernel: 3.8.0-35-generic
> vm os: ubuntu 12.04
> vm kernel: 3.2.0-56-virtual
> openstack: havana
> libvirt: 1.1.1-0ubuntu8~cloud2
>
> Thanks,
> Yuanle
>