CentOS virt - Feb 2014 - disk io in guests causes soft lockups in guests and host processes

Hi,
I have a strange phenomenon that I cannot readily explain so I wonder if 
anyone here can shed a light on this.

The host system is a Dell r815 with 64 cores and 256G ram and has centos 
6 installed. The five guests are also running centos 6 and are running 
as a hadoop cluster. The problem is that I see disk-io spikes in the 
vm's which then cause soft lockups in the guest but I also see hanging 
processes on the host as if the entire machine locks up for 30-60 seconds.

Now I know that having all cluster members running on the same system 
isn't efficient and that I cannot expect good performance but what I was 
not expecting is that a guest make host processes hang.
Does anyone have an idea what the issue could be here or how I can find 
out what cause for this behavior is?

Regards,
   Dennis

I experienced similar issues when disk images of virtual machines were 
stored in qcow/qcow2 files instead of logical volumes (LVM).
Using LVM gives you way better I/O performace than using qcow files.

Also very important: when you partition your disk drive(s) make sure 
that partitions are properly aligned to the physical allocation block 
size of the hard drive you use. Let's say your hard drive uses 4k 
sectors then every partition you create must start at a 4k boundary. If 
your partitions are mis-aligned then you'll get terrible disk I/O 
performace, just like the one you have described.
For more info see, for example: 
http://www.thomas-krenn.com/en/wiki/Partition_Alignment

You can also tweak Linux to get better KVM performance. For more info 
you can check out these documents:
http://pic.dhe.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaat/liaatbestpractices_pdf.pdf
http://www.novell.com/docrep/2013/05/kvm_virtualized_io_performance.pdf

Zoltan

On 2/20/2014 5:53 PM, Dennis Jacobfeuerborn wrote:
> Hi,
> I have a strange phenomenon that I cannot readily explain so I wonder if
> anyone here can shed a light on this.
>
> The host system is a Dell r815 with 64 cores and 256G ram and has centos
> 6 installed. The five guests are also running centos 6 and are running
> as a hadoop cluster. The problem is that I see disk-io spikes in the
> vm's which then cause soft lockups in the guest but I also see hanging
> processes on the host as if the entire machine locks up for 30-60 seconds.
>
> Now I know that having all cluster members running on the same system
> isn't efficient and that I cannot expect good performance but what I
was
> not expecting is that a guest make host processes hang.
> Does anyone have an idea what the issue could be here or how I can find
> out what cause for this behavior is?
>
> Regards,
>     Dennis
> _______________________________________________
> CentOS-virt mailing list
> CentOS-virt at centos.org
> http://lists.centos.org/mailman/listinfo/centos-virt
>