I have 2 similar servers. Since upgrading one from CentOS 5.5 to 6, disk write performance in kvm guest VMs is much worse. There are many, many posts about optimising kvm, many mentioning disk performance in CentOS 5 vs 6. I've tried various changes to speed up write performance, but northing's made a significant difference so far: - Install virtio disk drivers in guest - update the host software - Update RAID firmware to latest version - Switch the host disk scheduler to deadline - Increase host RAM from 8GB to 24GB - Increase guest RAM from 2GB to 4GB - Try different kvm cache options - Switch host from ext4 back to ext3 - Set noatime on the virtual disk image file Note: There is no encryption or on-access virus scanner on any host or guest. Below are some the block write figures in MB/s from bonnie++ with various configurations: First, figures for the hosts show that the CentOS 6 server is faster: 54 CentOS 5 Host 50 CentOS 5 Host 69 CentOS 6 host 70 CentOS 6 host Figures for a CentOS 6 guest running on the CentOS 5 host show that the performance hit is less than 50%: 30 CentOS 6 guest on CentOS 5 host with no optimisations 27 CentOS 6 guest on CentOS 5 host with no optimisations 32 CentOS 6 guest on CentOS 5 host with no optimisations Here are the figures a CentOS 6 guest running on the CentOS 6 host with various optimisations. Even with these optimisations, performance doesn't come close to the un-optimised guest running on the CentoOS 5 host: 5 No optimisations (i.e. same configuration as on CentOS 5) 4 deadline scheduler 5 deadline scheduler 15 noatime,nodiratime 14 noatime,nodiratime 15 noatime 15 noatime + deadline scheduler 13 virtio 13 virtio 10 virtio + noatime 9 virtio + noatime The CentOS 6 server has a better RAID card, different disks and more RAM, which might account for the better CentOS 6 host performance. But why might the guest write performance be so much worse? Is this a known problem? If so, what's the cause? If not, is there a way to locate the problem rather than using trial and error? Thanks, Julian -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos-virt/attachments/20120810/d7b4b057/attachment-0006.html>
On 10.08.2012 13:46, Julian price wrote:> I have 2 similar servers. Since upgrading one from CentOS 5.5 to 6, > disk write performance in kvm guest VMs is much worse. > > There are many, many posts about optimising kvm, many mentioning disk > performance in CentOS 5 vs 6. I've tried various changes to speed up > write performance, but northing's made a significant difference so > far: > > - Install virtio disk drivers in guest > - update the host software > - Update RAID firmware to latest version > - Switch the host disk scheduler to deadline > - Increase host RAM from 8GB to 24GB > - Increase guest RAM from 2GB to 4GB > - Try different kvm cache options > - Switch host from ext4 back to ext3 > - Set noatime on the virtual disk image file > Note: There is no encryption or on-access virus scanner on any host > or guest. > > Below are some the block write figures in MB/s from bonnie++ with > various configurations: > > First, figures for the hosts show that the CentOS 6 server is faster: > > 54 CentOS 5 Host > 50 CentOS 5 Host > 69 CentOS 6 host > 70 CentOS 6 host > > Figures for a CentOS 6 guest running on the CentOS 5 host show that > the performance hit is less than 50%: > > 30 CentOS 6 guest on CentOS 5 host with no optimisations > 27 CentOS 6 guest on CentOS 5 host with no optimisations > 32 CentOS 6 guest on CentOS 5 host with no optimisations > > Here are the figures a CentOS 6 guest running on the CentOS 6 host > with various optimisations. Even with these optimisations, > performance doesn't come close to the un-optimised guest running on > the CentoOS 5 host: > > 5 No optimisations (i.e. same configuration as on CentOS 5) > 4 deadline scheduler > 5 deadline scheduler > 15 noatime,nodiratime > 14 noatime,nodiratime > 15 noatime > 15 noatime + deadline scheduler > 13 virtio > 13 virtio > 10 virtio + noatime > 9 virtio + noatime > > The CentOS 6 server has a better RAID card, different disks and more > RAM, which might account for the better CentOS 6 host performance. > But why might the guest write performance be so much worse? > > Is this a known problem? If so, what's the cause? If not, is > there a way to locate the problem rather than using trial and error? > > Thanks, > JulianHow are you running KVM machines? Through libvirt? Have you made any changes to the cgroups? What are you using as disks, raw files, lvm? -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro
Nice post, Julian. It generated some feedback at http://irclog.perlgeek.de/crimsonfu/2012-08-10 and a link to http://rhsummit.files.wordpress.com/2012/03/wagner_network_perf.pdf Phil On Aug 10, 2012, at 8:46 AM, Julian price <centos.org at julianprice.org.uk> wrote:> I have 2 similar servers. Since upgrading one from CentOS 5.5 to 6, disk write performance in kvm guest VMs is much worse. > > There are many, many posts about optimising kvm, many mentioning disk performance in CentOS 5 vs 6. I've tried various changes to speed up write performance, but northing's made a significant difference so far: > > - Install virtio disk drivers in guest > - update the host software > - Update RAID firmware to latest version > - Switch the host disk scheduler to deadline > - Increase host RAM from 8GB to 24GB > - Increase guest RAM from 2GB to 4GB > - Try different kvm cache options > - Switch host from ext4 back to ext3 > - Set noatime on the virtual disk image file > Note: There is no encryption or on-access virus scanner on any host or guest. > > Below are some the block write figures in MB/s from bonnie++ with various configurations: > > First, figures for the hosts show that the CentOS 6 server is faster: > > 54 CentOS 5 Host > 50 CentOS 5 Host > 69 CentOS 6 host > 70 CentOS 6 host > > Figures for a CentOS 6 guest running on the CentOS 5 host show that the performance hit is less than 50%: > > 30 CentOS 6 guest on CentOS 5 host with no optimisations > 27 CentOS 6 guest on CentOS 5 host with no optimisations > 32 CentOS 6 guest on CentOS 5 host with no optimisations > > Here are the figures a CentOS 6 guest running on the CentOS 6 host with various optimisations. Even with these optimisations, performance doesn't come close to the un-optimised guest running on the CentoOS 5 host: > > 5 No optimisations (i.e. same configuration as on CentOS 5) > 4 deadline scheduler > 5 deadline scheduler > 15 noatime,nodiratime > 14 noatime,nodiratime > 15 noatime > 15 noatime + deadline scheduler > 13 virtio > 13 virtio > 10 virtio + noatime > 9 virtio + noatime > > The CentOS 6 server has a better RAID card, different disks and more RAM, which might account for the better CentOS 6 host performance. But why might the guest write performance be so much worse? > > Is this a known problem? If so, what's the cause? If not, is there a way to locate the problem rather than using trial and error? > > Thanks, > Julian > _______________________________________________ > CentOS-virt mailing list > CentOS-virt at centos.org > http://lists.centos.org/mailman/listinfo/centos-virt-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos-virt/attachments/20120811/35c1811e/attachment-0006.html>
>> I have 2 similar servers. Since upgrading one from CentOS 5.5 to 6,disk write performance in kvm guest VMs is much worse. Philip Durbin wrote: > Nice post, Julian. It generated some feedback at http://irclog.perlgeek.de/crimsonfu/2012-08-10 and a link to http://rhsummit.files.wordpress.com/2012/03/wagner_network_perf.pdf > > Phil Thanks Phil for linking to my post on #crimsonfu, and reporting the result back here. In response to the points jimi_c raised there: > I don't see his test numbers for different caching options All the test figures I gave are using cache=writethrough. cache=writeback produced much better figures than even the host (about 180MB/s), because it is really writing to memory, but I don't think it's a safe option to use. cache=none produced worse figures. I didn't include the figures because I did those tests before I started using bonnie++ (I was just timing copying files before that) and I'd already ruled cacheing out as a solution. > also, is he doing deadline on the guest, host, or both? deadline on the host - didn't try it on the guest. > not sure if they implemented it yet, but they were talking about a vm host setting for tuned > and one for guests > yeah, Wagner mentioned it in his summit presentation: http://rhsummit.files.wordpres[?]_network_perf.pdf > they should be available in rhel 6.3 according to his presentation Well, tuned-adm is a gift for part-time sysadmins like myself. Some of the guest disk write figures were close to the host's & better than CentOS 5 after doing... yum install tuned tuned-adm profile virtual-host ..in the host and... yum install tuned tuned-adm profile virtual-guest ...in the guest. Here are the new bonnie++ guest block write figures in MB/s. all using tuned-host and virtio, with & without tuned-guest. Not sure why there's so much variation, but at least they're all much better. 45 tuned-host 73 tuned-host 50 tuned-host + tuned-guest 37 tuned-host + tuned-guest > rhel/centos 6 probably went with a more conservative tuning option Certainly looks that way. It's be interesting to know what & why. Before jimi_c provided the tuned-adm tip, I was hoping that running the VM off a block device might be the answer. i.e: qemu-img convert /media/vm027/hda.raw -O /dev/vm/vm031 ...but they are worse than running off a raw virtual disk file. 16 tuned-host virtio 20 tuned-host virtio 27 tuned-host tuned-guest virtio 24 tuned-host tuned-guest virtio I'm not convinced, maybe there are other factors at work. I'd investigate further if my plan A wasn't back on track. The bonnie++ figures I gave before measuring host disk write performance were for the host *root* partition, not the LVM volumes that the guest VMs use. What if the problem is LVM, not KVM? So I did some timings comparing the root drive with LVM volumes, some with & some without tuned-host. 'archive' and 'vmxxx' are the lvm volume names: (Note: these timings are done in the host, not in a guest) 69 / 69 / 70 / 65 / + tuned-host 64 / + tuned-host 55 archive 56 archive 53 vm027 48 vm027 33 vm027 + tuned-host 38 vm027 + tuned-host 53 vm022 + tuned-host 85 vm022 + tuned-host This indicates that there is a wide variation in performance between different LVM volumes, and all the LVM volumes are performing worse than the root. (It's interesting that with tuned-host, the times seem to be mostly worse, but with greater deviation.) I repeated the above test on the CentOS 5 server (without tuned-host of course) and found the same thing - LVMs perform worse than root and vary widely: 54 / 50 / 39 archive 45 archive 39 archive 49 vm022 34 vm027 33 vm027 A slight performance hit might be expected for LVM, but I though it was meant to be negligible? If the figures fell into 2 bands - good and bad - then I'd be looking for a specific problem like a sector alignment, but they don't, and isn't sector alignment meant to be fixed on CentOS 6? The variation in performance indicates a problem of variable severity like fragmentation or the position on the physical disk - but I don't think either of those are likely causes, because there's only one file in each volume, and physical disk position shouldn't have such a marked effect should it? Any other suggestions? Thanks, Julian