Hi,
It looks like I am having problems when I create more than 100 VM in
one xen node.
up to 60 VMs there is zero problems and between 60 and 100 problems start
more than 100 the node becomes unstable.
The node has tons of memory (more than 30 GB free)
also the node has 16 cores so the processor isnt the issue here.
is there any limitation in xen that prevent the system from serving
more than 100 VMs ?
the symptoms are that
1- xm command become ultra slow and we are forced to restart the xend
demon regularly (xm create could take 5-20 min).
2- if we restart the node for any reason booting up all the VMs in
/etc/xen/auto will take 4 hours or more if we didnt restart the xend
demon in the middle.
3- the xend demon restart takes between 30 min up to 120 min.
4- after xend restart xm command bocomes fast for a while and start
slowing again as we issue more m commands
Dom-0 memory is 2 GB (i tried with 3 GB with no improvement at all)
xm info result
xm info
host : localhost.localdomain
release : 2.6.32.18
version : #1 SMP Wed Aug 25 11:17:42 EDT 2010
machine : x86_64
nr_cpus : 24
nr_nodes : 4
cores_per_socket : 12
threads_per_core : 1
cpu_mhz : 2100
hw_caps :
178bf3ff:efd3fbff:00000000:00001310:00802001:00000000:000837ff:00000000
virt_caps : hvm
total_memory : 131070
free_memory : 112496
node_to_cpu : node0:0-5
node1:6-11
node2:18-23
node3:12-17
node_to_memory : node0:20114
node1:32671
node2:29599
node3:30110
node_to_dma32_mem : node0:3514
node1:0
node2:0
node3:0
max_node_id : 3
xen_major : 4
xen_minor : 0
xen_extra : .1-rc5
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : Mon Aug 02 11:18:11 2010 +0100 21294:f200573325ba
xen_commandline : dom0_mem=12048M loglvl=all guest_loglvl=all
console=tty0 console=com2 com2=57600,8n1
cc_compiler : gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)
cc_compile_by : root
cc_compile_domain : localdomain
cc_compile_date : Sat Aug 21 23:10:21 EDT 2010
xend_config_format : 4
any help will be appreciated
Thanks
Can you try adding the xen command line options for increasing the xen heap space? http://support.citrix.com/article/CTX124259 There''s a formula in that article that explains how much would be the right amount. florian 2012/3/8 moftah moftah <mofta7y@gmail.com>:> Hi, > > It looks like I am having problems when I create more than 100 VM in > one xen node. > > up to 60 VMs there is zero problems and between 60 and 100 problems start > > more than 100 the node becomes unstable. > > The node has tons of memory (more than 30 GB free) > also the node has 16 cores so the processor isnt the issue here. > is there any limitation in xen that prevent the system from serving > more than 100 VMs ? > > the symptoms are that > 1- xm command become ultra slow and we are forced to restart the xend > demon regularly (xm create could take 5-20 min). > 2- if we restart the node for any reason booting up all the VMs in > /etc/xen/auto will take 4 hours or more if we didnt restart the xend > demon in the middle. > 3- the xend demon restart takes between 30 min up to 120 min. > 4- after xend restart xm command bocomes fast for a while and start > slowing again as we issue more m commands > > Dom-0 memory is 2 GB (i tried with 3 GB with no improvement at all) > > xm info result > xm info > host : localhost.localdomain > release : 2.6.32.18 > version : #1 SMP Wed Aug 25 11:17:42 EDT 2010 > machine : x86_64 > nr_cpus : 24 > nr_nodes : 4 > cores_per_socket : 12 > threads_per_core : 1 > cpu_mhz : 2100 > hw_caps : > 178bf3ff:efd3fbff:00000000:00001310:00802001:00000000:000837ff:00000000 > virt_caps : hvm > total_memory : 131070 > free_memory : 112496 > node_to_cpu : node0:0-5 > node1:6-11 > node2:18-23 > node3:12-17 > node_to_memory : node0:20114 > node1:32671 > node2:29599 > node3:30110 > node_to_dma32_mem : node0:3514 > node1:0 > node2:0 > node3:0 > max_node_id : 3 > xen_major : 4 > xen_minor : 0 > xen_extra : .1-rc5 > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 > hvm-3.0-x86_32p hvm-3.0-x86_64 > xen_scheduler : credit > xen_pagesize : 4096 > platform_params : virt_start=0xffff800000000000 > xen_changeset : Mon Aug 02 11:18:11 2010 +0100 21294:f200573325ba > xen_commandline : dom0_mem=12048M loglvl=all guest_loglvl=all > console=tty0 console=com2 com2=57600,8n1 > cc_compiler : gcc version 4.1.2 20080704 (Red Hat 4.1.2-48) > cc_compile_by : root > cc_compile_domain : localdomain > cc_compile_date : Sat Aug 21 23:10:21 EDT 2010 > xend_config_format : 4 > > > any help will be appreciated > > Thanks > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users-- the purpose of libvirt is to provide an abstraction layer hiding all xen features added since 2006 until they were finally understood and copied by the kvm devs.
Hello, Forgive the top post. - setup a host name HST=bigbertha echo "HOSTNAME=$HST">>/etc/sysconfig/network echo "127.0.1.1 $HST">>/etc/hosts - edit /etc/sysconfig/xendomains turn off save & restore - in xen_commandline : dom0_mem=12048M loglvl=all guest_loglvl=all ^^ too high in your grub cfg, edit xen.gz cmdline and set dom0_mem=2048M <--- 2G for dom0 try those 3 changes. -- Mark ----- Original Message ----- From: moftah moftah <mofta7y@gmail.com> To: xen-users@lists.xen.org Cc: Sent: Thursday, March 8, 2012 11:05 AM Subject: [Xen-users] Maximum number of VMs per xen server Hi, It looks like I am having problems when I create more than 100 VM in one xen node. up to 60 VMs there is zero problems and between 60 and 100 problems start more than 100 the node becomes unstable. The node has tons of memory (more than 30 GB free) also the node has 16 cores so the processor isnt the issue here. is there any limitation in xen that prevent the system from serving more than 100 VMs ? the symptoms are that 1- xm command become ultra slow and we are forced to restart the xend demon regularly (xm create could take 5-20 min). 2- if we restart the node for any reason booting up all the VMs in /etc/xen/auto will take 4 hours or more if we didnt restart the xend demon in the middle. 3- the xend demon restart takes between 30 min up to 120 min. 4- after xend restart xm command bocomes fast for a while and start slowing again as we issue more m commands Dom-0 memory is 2 GB (i tried with 3 GB with no improvement at all) xm info result xm info host : localhost.localdomain release : 2.6.32.18 version : #1 SMP Wed Aug 25 11:17:42 EDT 2010 machine : x86_64 nr_cpus : 24 nr_nodes : 4 cores_per_socket : 12 threads_per_core : 1 cpu_mhz : 2100 hw_caps : 178bf3ff:efd3fbff:00000000:00001310:00802001:00000000:000837ff:00000000 virt_caps : hvm total_memory : 131070 free_memory : 112496 node_to_cpu : node0:0-5 node1:6-11 node2:18-23 node3:12-17 node_to_memory : node0:20114 node1:32671 node2:29599 node3:30110 node_to_dma32_mem : node0:3514 node1:0 node2:0 node3:0 max_node_id : 3 xen_major : 4 xen_minor : 0 xen_extra : .1-rc5 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : Mon Aug 02 11:18:11 2010 +0100 21294:f200573325ba xen_commandline : dom0_mem=12048M loglvl=all guest_loglvl=all console=tty0 console=com2 com2=57600,8n1 cc_compiler : gcc version 4.1.2 20080704 (Red Hat 4.1.2-48) cc_compile_by : root cc_compile_domain : localdomain cc_compile_date : Sat Aug 21 23:10:21 EDT 2010 xend_config_format : 4 any help will be appreciated Thanks _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Hi Mark, for the second point what do you want me to edit in /etc/sysconfig/xendomains I can turn it off but will that affect the running VMs ? yes actually it was at 2GB but the server restarted while I was writing the email !! so i decided to give it huge value just to keep it up until someone can help me On 12-03-08 02:57 PM, Mark Pryor wrote:> Hello, > > Forgive the top post. > > - setup a host name > HST=bigbertha > echo "HOSTNAME=$HST">>/etc/sysconfig/network > echo "127.0.1.1 $HST">>/etc/hosts > > - edit /etc/sysconfig/xendomains > turn off save& restore > > - in xen_commandline : dom0_mem=12048M loglvl=all guest_loglvl=all > > ^^ too high > in your grub cfg, edit xen.gz cmdline and set > dom0_mem=2048M<--- 2G for dom0 > > try those 3 changes. >
The server restarted by itself as I have expected. currently I have dom0 with 4 GB of RAM only and I have set the xenheap_megabytes=52 it does not look like that helped because xend became so slow stating up VM # 50 and I was forced to restart xend I am restarting it now (it took already 24 minutes) Thanks On 12-03-08 02:57 PM, Mark Pryor wrote:> Hello, > > Forgive the top post. > > - setup a host name > HST=bigbertha > echo "HOSTNAME=$HST">>/etc/sysconfig/network > echo "127.0.1.1 $HST">>/etc/hosts > > - edit /etc/sysconfig/xendomains > turn off save& restore > > - in xen_commandline : dom0_mem=12048M loglvl=all guest_loglvl=all > > ^^ too high > in your grub cfg, edit xen.gz cmdline and set > dom0_mem=2048M<--- 2G for dom0 > > try those 3 changes. >
Hi all Have anyone tried running more than 100 VMs on the same node successfully ? if yes is it possible to share with us your xen version and what tricks you used to resolve the issue ? Thanks
On Mar 9, 2012 2:13 AM, "moftah moftah" <mofta7y@gmail.com> wrote:> > Hi, > > It looks like I am having problems when I create more than 100 VM in > one xen node. > > up to 60 VMs there is zero problems and between 60 and 100 problems start > > more than 100 the node becomes unstable. > > The node has tons of memory (more than 30 GB free) > also the node has 16 cores so the processor isnt the issue here. > is there any limitation in xen that prevent the system from serving > more than 100 VMs ? >You''re pushing 6 VMs per core, if the VMs are not paravirtualized, that''s a huge load. There''s also the question of I/O bottleneck. After starting the 60th VM, i.e., when slowdowns begin, how''s the processor and I/O load? Rgds, _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Hi Pandu, thanks for your answer, really appreciate it. About 70% of the VMs are paravirtualized Linux. The remaining 30% are Windows with GPLPV drivers. when reaching the 60th VM the CPU utilization will reach 2 - 3 on Dom0 which has access to 12 cores (so the utilization is approx 25%) for Harddisk access I didn''t check at that moment. but if the issue is related tot he resources (IO or CPU) then even after I restart xend the problem should remain. However when I restart xend (takes 30 min) the server can boot remaining VMs very fast (start slowing again when reaching above 50 new VMs) so the issue looks like xend issue as it cannot start up more than 60 VMs without a long restart and after the restart I can start 60 VMs more and so on Thanks On 12-03-08 10:04 PM, Pandu Poluan wrote:> > > On Mar 9, 2012 2:13 AM, "moftah moftah" <mofta7y@gmail.com > <mailto:mofta7y@gmail.com>> wrote: > > > > Hi, > > > > It looks like I am having problems when I create more than 100 VM in > > one xen node. > > > > up to 60 VMs there is zero problems and between 60 and 100 problems > start > > > > more than 100 the node becomes unstable. > > > > The node has tons of memory (more than 30 GB free) > > also the node has 16 cores so the processor isnt the issue here. > > is there any limitation in xen that prevent the system from serving > > more than 100 VMs ? > > > > You''re pushing 6 VMs per core, if the VMs are not paravirtualized, > that''s a huge load. There''s also the question of I/O bottleneck. > > After starting the 60th VM, i.e., when slowdowns begin, how''s the > processor and I/O load? > > Rgds, >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users