libvirt users - Sep 2012 - Kernel unresponsive after booting 700+ vm's on a single host

For a research project we are trying to boot a very large amount of tiny, custom
built VM's on KVM/ubuntu. The maximum VM-count achieved was 1000, but with
substantial slowness, and eventually kernel failure, while the cpu/memory loads
were nowhere near limits. Where is the likely bottleneck? Any solutions,
workarounds, hacks or dirty tricks?

A few more details here (tumbleweed question), and the possibility of an upvote:
http://stackoverflow.com/questions/12243129/kvm-qemu-maximum-vm-count-limit

Any tips would be much appreciated!

Best regards,

Alfred Bratterud
Assistant Professor
Dept. of Computer Science,
Oslo and Akershus University College
of Applied Sciences
P:  (+47) 2245 3263
M: (+47) 4102 0222

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20120910/62f4aa6c/attachment.htm>

On 09/10/2012 07:51 AM, Alfred Bratterud wrote:
> For a research project we are trying to boot a very large amount of tiny,
custom built VM's on KVM/ubuntu. The maximum VM-count achieved was 1000, but
with substantial slowness, and eventually kernel failure, while the cpu/memory
loads were nowhere near limits. Where is the likely bottleneck? Any solutions,
workarounds, hacks or dirty tricks?
Are you using cgroups?  There have been some known bottlenecks in the
kernel cgroup code, where it scales very miserably; and since libvirt
uses a different cgroup per VM by default when cgroups are enabled, that
might explain part of the problem.

Other than that, if you can profile the slowdowns, I'm sure people would
be interested in the results.

-- 
Eric Blake   eblake at redhat.com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 617 bytes
Desc: OpenPGP digital signature
URL:
<http://listman.redhat.com/archives/libvirt-users/attachments/20120910/23273202/attachment.sig>