Mathew Crane
2018-Sep-11 14:50 UTC
[libvirt-users] Secondary disk unattached if guest unresponsive and killed with 'virsh destroy'
I have a very strange issue that has plagued our KVM-based hypervisor infrastructure. Environment: - All EL7-based libvirt releases since EL7.0 - Fully patched EL7-based KVM guest on a fully patched EL7-based KVM host. - Secondary disk attached via 'virsh attach-device guest /vm/guest-with-secondary-disk-01/disk2.xml --persistent' (note --persistent flag here) Problem Description: - Guest becomes unresponsive/hangs due to issue on guest. Typically the scenario is that PID1 hangs or goes into a bad state due to a misbehaving application. - Before virsh destroy, 'virsh dumpxml guest | grep vd[ab]' returns the expected filesystems. - Guest is destroyed from the hypervisor via 'virsh destroy guest'. - With guest still powered off, 'virsh dumpxml guest | grep vd[ab]' only shows 'vda'. 'vdb' now unattached. - To fix, requires re-run of 'virsh attach-device guest guest-with-secondary-disk-01/disk2.xml --persistent' Expected Behavior: - 'virsh destroy guest' does not drop the secondary disk attached to the guest if the node is in a hung state. Other notes: - If guest is not in a hung state, 'virsh destroy' does not drop the secondary disk. - 'virsh reset guest' does not exhibit the same behavior when guest is hung - vdb remains attached. - No logs produced in /var/log/libvirt indicating an issue, even with debug logging enabled. Is there any obvious I'm missing? Is this expected behavior? Guest XML via 'virsh dumpxml guest': <domain type='kvm' id='17'> <name>guest-with-secondary-disk-01</name> <uuid>ab578b51-3afd-4d01-86ab-a16b1b953919</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>1</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type> <boot dev='hd'/> <bootmenu enable='yes'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/vm/guest-with-secondary-disk-01/guest-with-secondary-disk-01.img'/> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/vm/guest-with-secondary-disk-01/guest-with-secondary-disk-01_2.img'/> <backingStore/> <target dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk> <controller type='usb' index='0' model='piix3-uhci'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='bridge'> <mac address='01:02:03:99:99:99'/> <source bridge='bridge_99'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='eth0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </interface> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/guest-with-secondary-disk-01.agent'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1' keymap='en-us'> <listen type='address' address='127.0.0.1'/> </graphics> <video> <model type='cirrus' vram='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </memballoon> <rng model='virtio'> <backend model='random'>/dev/random</backend> <alias name='rng0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </rng> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c259,c495</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c259,c495</imagelabel> </seclabel> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+107:+107</label> <imagelabel>+107:+107</imagelabel> </seclabel> </domain> Secondary disk XML: <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/vm/guest-with-secondary-disk-01/guest-with-secondary-disk-01_2.img'/> <target dev='vdb'/> </disk>