Hi,
I'm investigating a performance issue on a virtualized windows server
host that is run on a ubuntu machine via libvirt/qemu. While the host
can easily read/write on the raid drive with 100Mmb/s as observable with
dd, the virtualized windows server running on that host is barely able
to read/write with at most 8Mb/s and averages around 1.4Mb/s. This has
grown to the extent that the virtualized host is often unresponsive and
even unable to start up its services with system default timeouts. Any
help to improve the situation is greatly appreciated.
This is the configuration of the virtualized host:
~$ virsh dumpxml windows-server-2016-x64
<domain type='kvm' id='1'>
<name>windows-server-2016-x64</name>
<uuid>XXX</uuid>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<vcpu placement='static'>2</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64'
machine='pc-i440fx-xenial'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
</hyperv>
</features>
<cpu mode='custom' match='exact'>
<model fallback='allow'>IvyBridge</model>
<topology sockets='1' cores='2' threads='1'/>
</cpu>
<clock offset='localtime'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'
cache='none'/>
<source
file='/var/data/virtuals/machines/windows-server-2016-x64/image.qcow2'/>
<backingStore/>
<target dev='hda' bus='ide'/>
<alias name='ide0-0-0'/>
<address type='drive' controller='0' bus='0'
target='0' unit='0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source
file='/var/data/virtuals/machines/windows-server-2016-x64/dvd.iso'/>
<backingStore/>
<target dev='hdb' bus='ide'/>
<readonly/>
<alias name='ide0-0-1'/>
<address type='drive' controller='0' bus='0'
target='0' unit='1'/>
</disk>
<controller type='usb' index='0'
model='ich9-ehci1'>
<alias name='usb'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x05'
function='0x7'/>
</controller>
<controller type='usb' index='0'
model='ich9-uhci1'>
<alias name='usb'/>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x05'
function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0'
model='ich9-uhci2'>
<alias name='usb'/>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x05'
function='0x1'/>
</controller>
<controller type='usb' index='0'
model='ich9-uhci3'>
<alias name='usb'/>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x05'
function='0x2'/>
</controller>
<controller type='pci' index='0'
model='pci-root'>
<alias name='pci.0'/>
</controller>
<controller type='ide' index='0'>
<alias name='ide'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x01'
function='0x1'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:0e:f2:23'/>
<source bridge='br0'/>
<target dev='vnet0'/>
<model type='rtl8139'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03'
function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/1'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/1'>
<source path='/dev/pts/1'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<input type='tablet' bus='usb'>
<alias name='input0'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='5900' autoport='yes'
listen='127.0.0.1'>
<listen type='address' address='127.0.0.1'/>
</graphics>
<video>
<model type='vga' vram='16384' heads='1'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x04'
function='0x0'/>
</memballoon>
</devices>
<seclabel type='dynamic' model='apparmor'
relabel='yes'>
<label>libvirt-XXX</label>
<imagelabel>libvirt-XXX</imagelabel>
</seclabel>
</domain>
Cheers,
Dominik
Hi Dominik, Sure, I beleive you can improve using: <cpu mode='host-passthrough'> </cpu> add io='native' <driver name='qemu' type='qcow2' cache='none' io='native'/> After that, please try again, but I can see other thing, for example, change the hda=IDE to virtio. Cheers! Thiago 2017-06-14 5:26 GMT-03:00 Dominik Psenner <dpsenner@gmail.com>:> Hi, > > I'm investigating a performance issue on a virtualized windows server host > that is run on a ubuntu machine via libvirt/qemu. While the host can easily > read/write on the raid drive with 100Mmb/s as observable with dd, the > virtualized windows server running on that host is barely able to > read/write with at most 8Mb/s and averages around 1.4Mb/s. This has grown > to the extent that the virtualized host is often unresponsive and even > unable to start up its services with system default timeouts. Any help to > improve the situation is greatly appreciated. > > This is the configuration of the virtualized host: > > ~$ virsh dumpxml windows-server-2016-x64 > <domain type='kvm' id='1'> > <name>windows-server-2016-x64</name> > <uuid>XXX</uuid> > <memory unit='KiB'>2097152</memory> > <currentMemory unit='KiB'>2097152</currentMemory> > <vcpu placement='static'>2</vcpu> > <resource> > <partition>/machine</partition> > </resource> > <os> > <type arch='x86_64' machine='pc-i440fx-xenial'>hvm</type> > <boot dev='hd'/> > </os> > <features> > <acpi/> > <apic/> > <hyperv> > <relaxed state='on'/> > <vapic state='on'/> > <spinlocks state='on' retries='8191'/> > </hyperv> > </features> > <cpu mode='custom' match='exact'> > <model fallback='allow'>IvyBridge</model> > <topology sockets='1' cores='2' threads='1'/> > </cpu> > <clock offset='localtime'> > <timer name='rtc' tickpolicy='catchup'/> > <timer name='pit' tickpolicy='delay'/> > <timer name='hpet' present='no'/> > <timer name='hypervclock' present='yes'/> > </clock> > <on_poweroff>destroy</on_poweroff> > <on_reboot>restart</on_reboot> > <on_crash>restart</on_crash> > <pm> > <suspend-to-mem enabled='no'/> > <suspend-to-disk enabled='no'/> > </pm> > <devices> > <emulator>/usr/bin/kvm-spice</emulator> > <disk type='file' device='disk'> > <driver name='qemu' type='qcow2' cache='none'/> > <source file='/var/data/virtuals/machines/windows-server-2016-x64/ > image.qcow2'/> > <backingStore/> > <target dev='hda' bus='ide'/> > <alias name='ide0-0-0'/> > <address type='drive' controller='0' bus='0' target='0' unit='0'/> > </disk> > <disk type='file' device='cdrom'> > <driver name='qemu' type='raw'/> > <source file='/var/data/virtuals/machines/windows-server-2016-x64/ > dvd.iso'/> > <backingStore/> > <target dev='hdb' bus='ide'/> > <readonly/> > <alias name='ide0-0-1'/> > <address type='drive' controller='0' bus='0' target='0' unit='1'/> > </disk> > <controller type='usb' index='0' model='ich9-ehci1'> > <alias name='usb'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x05' > function='0x7'/> > </controller> > <controller type='usb' index='0' model='ich9-uhci1'> > <alias name='usb'/> > <master startport='0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x05' > function='0x0' multifunction='on'/> > </controller> > <controller type='usb' index='0' model='ich9-uhci2'> > <alias name='usb'/> > <master startport='2'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x05' > function='0x1'/> > </controller> > <controller type='usb' index='0' model='ich9-uhci3'> > <alias name='usb'/> > <master startport='4'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x05' > function='0x2'/> > </controller> > <controller type='pci' index='0' model='pci-root'> > <alias name='pci.0'/> > </controller> > <controller type='ide' index='0'> > <alias name='ide'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' > function='0x1'/> > </controller> > <interface type='bridge'> > <mac address='52:54:00:0e:f2:23'/> > <source bridge='br0'/> > <target dev='vnet0'/> > <model type='rtl8139'/> > <alias name='net0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' > function='0x0'/> > </interface> > <serial type='pty'> > <source path='/dev/pts/1'/> > <target port='0'/> > <alias name='serial0'/> > </serial> > <console type='pty' tty='/dev/pts/1'> > <source path='/dev/pts/1'/> > <target type='serial' port='0'/> > <alias name='serial0'/> > </console> > <input type='tablet' bus='usb'> > <alias name='input0'/> > </input> > <input type='mouse' bus='ps2'/> > <input type='keyboard' bus='ps2'/> > <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'> > <listen type='address' address='127.0.0.1'/> > </graphics> > <video> > <model type='vga' vram='16384' heads='1'/> > <alias name='video0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > function='0x0'/> > </video> > <memballoon model='virtio'> > <alias name='balloon0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x04' > function='0x0'/> > </memballoon> > </devices> > <seclabel type='dynamic' model='apparmor' relabel='yes'> > <label>libvirt-XXX</label> > <imagelabel>libvirt-XXX</imagelabel> > </seclabel> > </domain> > > Cheers, > Dominik > > _______________________________________________ > libvirt-users mailing list > libvirt-users@redhat.com > https://www.redhat.com/mailman/listinfo/libvirt-users >
On Wed, 2017-06-14 at 15:32 -0300, Thiago Oliveira wrote: [...]> I can see other thing, for example, change the hda=IDE to virtio.I'd say switching the disk from IDE to virtio should be the very first step - and while you're at it, you might as well use virtio for the network interface too. -- Andrea Bolognani / Red Hat / Virtualization
On Wed, Jun 14, 2017 at 10:26:09AM +0200, Dominik Psenner wrote:> Hi, > > I'm investigating a performance issue on a virtualized windows server host > that is run on a ubuntu machine via libvirt/qemu. While the host can easily > read/write on the raid drive with 100Mmb/s as observable with dd, the > virtualized windows server running on that host is barely able to read/write > with at most 8Mb/s and averages around 1.4Mb/s. This has grown to the extent > that the virtualized host is often unresponsive and even unable to start up > its services with system default timeouts. Any help to improve the situation > is greatly appreciated. >Just to provide some even weirder numbers. With Debian on bare metal hosting Debian VM by qemu, kvm, for the I/O model comparison, I found there were about 40% performance drop (read), 26K(transaction/s) to 15 K (transaction/s) in pbench switching from IDE to virtio; while 18 K for bare metal. There would be 0 % to 70 % improvement in apache bench (request/s) depending on concurrency level, switching from IDE-e1000 to virtio-virtio. Very likely something was seriously wrong, as I expected when I/O bound, virtio would have been the winner. Dan> This is the configuration of the virtualized host: > > ~$ virsh dumpxml windows-server-2016-x64 > <domain type='kvm' id='1'> > <name>windows-server-2016-x64</name> > <uuid>XXX</uuid> > <memory unit='KiB'>2097152</memory> > <currentMemory unit='KiB'>2097152</currentMemory> > <vcpu placement='static'>2</vcpu> > <resource> > <partition>/machine</partition> > </resource> > <os> > <type arch='x86_64' machine='pc-i440fx-xenial'>hvm</type> > <boot dev='hd'/> > </os> > <features> > <acpi/> > <apic/> > <hyperv> > <relaxed state='on'/> > <vapic state='on'/> > <spinlocks state='on' retries='8191'/> > </hyperv> > </features> > <cpu mode='custom' match='exact'> > <model fallback='allow'>IvyBridge</model> > <topology sockets='1' cores='2' threads='1'/> > </cpu> > <clock offset='localtime'> > <timer name='rtc' tickpolicy='catchup'/> > <timer name='pit' tickpolicy='delay'/> > <timer name='hpet' present='no'/> > <timer name='hypervclock' present='yes'/> > </clock> > <on_poweroff>destroy</on_poweroff> > <on_reboot>restart</on_reboot> > <on_crash>restart</on_crash> > <pm> > <suspend-to-mem enabled='no'/> > <suspend-to-disk enabled='no'/> > </pm> > <devices> > <emulator>/usr/bin/kvm-spice</emulator> > <disk type='file' device='disk'> > <driver name='qemu' type='qcow2' cache='none'/> > <source > file='/var/data/virtuals/machines/windows-server-2016-x64/image.qcow2'/> > <backingStore/> > <target dev='hda' bus='ide'/> > <alias name='ide0-0-0'/> > <address type='drive' controller='0' bus='0' target='0' unit='0'/> > </disk> > <disk type='file' device='cdrom'> > <driver name='qemu' type='raw'/> > <source > file='/var/data/virtuals/machines/windows-server-2016-x64/dvd.iso'/> > <backingStore/> > <target dev='hdb' bus='ide'/> > <readonly/> > <alias name='ide0-0-1'/> > <address type='drive' controller='0' bus='0' target='0' unit='1'/> > </disk> > <controller type='usb' index='0' model='ich9-ehci1'> > <alias name='usb'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x05' > function='0x7'/> > </controller> > <controller type='usb' index='0' model='ich9-uhci1'> > <alias name='usb'/> > <master startport='0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x05' > function='0x0' multifunction='on'/> > </controller> > <controller type='usb' index='0' model='ich9-uhci2'> > <alias name='usb'/> > <master startport='2'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x05' > function='0x1'/> > </controller> > <controller type='usb' index='0' model='ich9-uhci3'> > <alias name='usb'/> > <master startport='4'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x05' > function='0x2'/> > </controller> > <controller type='pci' index='0' model='pci-root'> > <alias name='pci.0'/> > </controller> > <controller type='ide' index='0'> > <alias name='ide'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' > function='0x1'/> > </controller> > <interface type='bridge'> > <mac address='52:54:00:0e:f2:23'/> > <source bridge='br0'/> > <target dev='vnet0'/> > <model type='rtl8139'/> > <alias name='net0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' > function='0x0'/> > </interface> > <serial type='pty'> > <source path='/dev/pts/1'/> > <target port='0'/> > <alias name='serial0'/> > </serial> > <console type='pty' tty='/dev/pts/1'> > <source path='/dev/pts/1'/> > <target type='serial' port='0'/> > <alias name='serial0'/> > </console> > <input type='tablet' bus='usb'> > <alias name='input0'/> > </input> > <input type='mouse' bus='ps2'/> > <input type='keyboard' bus='ps2'/> > <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'> > <listen type='address' address='127.0.0.1'/> > </graphics> > <video> > <model type='vga' vram='16384' heads='1'/> > <alias name='video0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > function='0x0'/> > </video> > <memballoon model='virtio'> > <alias name='balloon0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x04' > function='0x0'/> > </memballoon> > </devices> > <seclabel type='dynamic' model='apparmor' relabel='yes'> > <label>libvirt-XXX</label> > <imagelabel>libvirt-XXX</imagelabel> > </seclabel> > </domain> > > Cheers, > Dominik > > _______________________________________________ > libvirt-users mailing list > libvirt-users@redhat.com > https://www.redhat.com/mailman/listinfo/libvirt-users