Xen users - Jul 2010 - XCP - extreme high load on pool master

Good day. 

We trying to test XCP cloud (4 hosts, each with 24Gb mem and 8 cores)
with some product like load...

But with just about 30-40 virtual machines I got an extreme load on dom0
on pool master host: LA is about 5.5-6, and most time are used by xapi
and stunnel processes.

It''s really bother me: what happens on higher load with few thousands
of
VMs with about 10-16 hosts in pool...

top data:

Tasks:  95 total,   3 running,  91 sleeping,   0 stopped,   1 zombie
Cpu(s):19.4%us,42.1%sy,0.0%ni,35.8%id,  1.3%wa,  0.0%hi,  1.0%si, 0.3%st
Mem:    746496k total,   731256k used,    15240k free,    31372k buffers
Swap:   524280k total,      128k used,   524152k free,   498872k cached

PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ COMMAND
17  -3  315m  14m 5844 S 52.3  2.0   5370:41 xapi
17  -3 22524  15m 1192 S  8.3  2.2 875:16.18 stunnel
15  -5     0    0    0 S  0.7  0.0  54:28.78 netback
10 -10  6384 1868  892 S  0.3  0.3  22:03.14 ovs-vswitchd

dom0 on non-master hosts are loaded about 30% each.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi George,

When xapi is doing work it normally writes large amounts of log-spam to
/var/log/xensource.log explaining what it's doing. Could you have a look and
see (i) at what rate it is writing; and (ii) what kind of things it is writing
about? If there's nothing in the logs it would be worth capturing a
'strace'.

We try to ensure that xapi is idle when there's no management activity going
on.

Another possibility is that the dom0 kernel is busy doing work that isn't
getting accounted to a separate task but is instead being accounted to existing
tasks. This happened once before (on an older kernel version) and it caused xapi
to look artificially very busy.

Cheers,
Dave
> -----Original Message-----
> From: xen-users-bounces@lists.xensource.com [mailto:xen-users-
> bounces@lists.xensource.com] On Behalf Of George Shuklin
> Sent: 20 July 2010 22:52
> To: xen-users@lists.xensource.com
> Subject: [Xen-users] XCP - extreme high load on pool master
> 
> Good day.
> 
> We trying to test XCP cloud (4 hosts, each with 24Gb mem and 8 cores)
> with some product like load...
> 
> But with just about 30-40 virtual machines I got an extreme load on
> dom0
> on pool master host: LA is about 5.5-6, and most time are used by xapi
> and stunnel processes.
> 
> It's really bother me: what happens on higher load with few thousands
> of
> VMs with about 10-16 hosts in pool...
> 
> top data:
> 
> Tasks:  95 total,   3 running,  91 sleeping,   0 stopped,   1 zombie
> Cpu(s):19.4%us,42.1%sy,0.0%ni,35.8%id,  1.3%wa,  0.0%hi,  1.0%si,
> 0.3%st
> Mem:    746496k total,   731256k used,    15240k free,    31372k
> buffers
> Swap:   524280k total,      128k used,   524152k free,   498872k cached
> 
> PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ COMMAND
> 17  -3  315m  14m 5844 S 52.3  2.0   5370:41 xapi
> 17  -3 22524  15m 1192 S  8.3  2.2 875:16.18 stunnel
> 15  -5     0    0    0 S  0.7  0.0  54:28.78 netback
> 10 -10  6384 1868  892 S  0.3  0.3  22:03.14 ovs-vswitchd
> 
> dom0 on non-master hosts are loaded about 30% each.
> 
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users