Hello all,
I''ve been having a very odd problem the last few days and I''m
at a loss
on how to proceed, so I thought I''d ask the user community.
I''m hitting
what *appears* to be some kind of wall on either the number of vm''s or
the amount of ram they''re using (personally I think it''s ram,
but I
don''t see how).
We have two identical CentOS 5.4 xen servers (xen-3.0.3-94.el5_4.2) in
production with 7 vm''s and dom0 on each server. I''ll post the
full
config on the bottom of the email, but here are the basics.
xentop - 13:58:32 Xen 3.1.2-164.6.1.el5
8 domains: 1 running, 7 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
Mem: 16776420k total, 5582324k used, 11194096k free CPUs: 4 @ 3000MHz
dom0_mem=1g
loops = 64
Now, if I attempt to install an 8th vm with 512MB or ram on either
server the install runs for awhile and then suddenly load on the xen
server begins to grow and grow and grow. Going from a load average:
0.09, 0.15, 0.10 to load average: 10.58, 7.88, 4.70. The box itself
begin to become very slow and bogged down. top shows that it''s because
of cpu iowait, but I can''t find any particular reason for it.
top - 13:35:18 up 63 days, 13:54, 3 users, load average: 10.58, 7.88, 4.70
Tasks: 196 total, 1 running, 195 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 24.9%id, 74.5%wa, 0.0%hi, 0.0%si,
0.7%st
75% is ridiculously high and it continues to get worse until the install
finally barely finishes after much longer than normal. An average
install takes 20 minutes, they''re taking around 35-40 now and I fear
that if I add yet another vm it will be even worse. Once the install
finishes the box returns to normal and I can boot the new vm normally as
well.
If I shutdown one of the existing vm''s using 512MB of ram, then the
install works perfectly and the load never goes crazy. Turn the 7th back
on , and installing the 8th is horrible again. I can duplicate this on
both servers which is very weird, but I can''t figure out *why*
it''s
happening. The fact that both servers have the problem leads me to
believe its a config problem.
I''m leaning towards something running out of ram and swapping hard,
which is why load takes awhile to start climbing and the cpu wait gets
very high, yet I have *more* than enough memory allocated to everything.
The installs are very io intensive, but I can''t understand why that 8th
vm is causing hell.
Is there some weird limit to the amount of vm''s or ram I can allocate?
Does dom0 need more than 1g and that''s why it''s getting
cranky? Is there
a config option I''m not aware of?
Any help would be appreciated, this is really worrying me. Thanks.
Swap:
# free
total used free shared buffers cached
Mem: 1048576 1039176 9400 0 10924 777816
-/+ buffers/cache: 250436 798140
Swap: 10482404 188 10482216
Full config:
# xm info
host : xen2.rutgers.edu
release : 2.6.18-164.6.1.el5xen
version : #1 SMP Tue Nov 3 16:48:13 EST 2009
machine : x86_64
nr_cpus : 4
nr_nodes : 1
sockets_per_node : 2
cores_per_socket : 2
threads_per_core : 1
cpu_mhz : 3000
hw_caps :
178bfbff:ebd3fbff:00000000:00000010:00002001:00000000:0000001f
total_memory : 16383
free_memory : 10931
node_to_cpu : node0:0-3
xen_major : 3
xen_minor : 1
xen_extra : .2-164.6.1.el5
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : unavailable
cc_compiler : gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
cc_compile_by : mockbuild
cc_compile_domain : centos.org
cc_compile_date : Tue Nov 3 16:04:14 EST 2009
xend_config_format : 2
--
===============================David Halik
System Administrator
OIT-CSS Rutgers University
dhalik@jla.rutgers.edu
===============================
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users