I have a question concerning the CPU utilization number provided by
the ''xm top'' command in xen 3.0. I''ve noticed that
if I run a TCP
performance tool on domain 0 which simply receives and echos back
a TCP stream, the output of ''xm top'' during the test does not
match
up with the output of ''top'' on domain 0 in terms of CPU
utilization
(nor with the CPU utilization computed by the test program).
In this example, ''xm top'' shows a CPU utilization for domain 0
of
around 65% during the test while ''top'' at the same time
indicates
that the domain is around 70% idle (the test tool also indicates
that the CPU utilization averages about 30% during the test).
Output from ''xm top'':
xentop - 12:44:29 Xen 3.0.0
1 domains: 1 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0
shutdown
Mem: 2094904k total, 559792k used, 1535112k free CPUs: 2 @ 3800MHz
NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k)
MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) SSID
Domain-0 -----r 273 66.6 519192 24.8 no limit
n/a 2 8 17120030 17489822 0
VCPUs(sec): 0: 273s
Output from ''top'':
Tasks: 69 total, 2 running, 67 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.0% us, 7.3% sy, 0.0% ni, 73.3% id, 0.0% wa, 3.3% hi,
15.0% si
Mem: 505556k total, 497812k used, 7744k free, 109624k buffers
Swap: 2104472k total, 0k used, 2104472k free, 212904k cached
Any idea why these displays disagree?
Thanks,
Dave Thompson
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Dave Thompson (davetho) wrote:>I have a question concerning the CPU utilization number provided by >the ''xm top'' command in xen 3.0. I''ve noticed that if I run a TCP >performance tool on domain 0 which simply receives and echos back >a TCP stream, the output of ''xm top'' during the test does not match >up with the output of ''top'' on domain 0 in terms of CPU utilization >(nor with the CPU utilization computed by the test program). > >... > >Any idea why these displays disagree? > >Top shows cpu usage for user processes in dom0, while xm top shows cpu usage for the domain, which includes cpu time spent in the dom0 kernel. Perhaps the thing you''re running incurs lots of kernel cpu time, but not user time? Rob _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> -----Original Message----- > From: Rob Gardner [mailto:rob.gardner@hp.com] > Sent: Monday, December 12, 2005 1:23 PM > To: Dave Thompson (davetho) > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] CPU Utilization > > Dave Thompson (davetho) wrote: > > >I have a question concerning the CPU utilization number provided by > >the ''xm top'' command in xen 3.0. I''ve noticed that if I run a TCP > >performance tool on domain 0 which simply receives and echos back > >a TCP stream, the output of ''xm top'' during the test does not match > >up with the output of ''top'' on domain 0 in terms of CPU utilization > >(nor with the CPU utilization computed by the test program). > > > >... > > > >Any idea why these displays disagree? > > > > > > Top shows cpu usage for user processes in dom0, while xm top > shows cpu usage for the domain, which includes cpu time spent > in the dom0 kernel.Doesn''t top''s CPU summary line show both? I thought us=user space, sy=system, ni=nice, id=idle, hi=hwintr, si=softintr. So in my example 1% in user space, 7.3% in system level, 3.3% in h/w intr, and 15% in soft intr. Total of 26.6% utilized of which 25.6% is in kernel.> Perhaps the thing you''re running incurs lots of kernel cpu > time, but not > user time? > > Rob >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dave Thompson (davetho) wrote:> >Doesn''t top''s CPU summary line show both? I thought us=user space, >sy=system, ni=nice, id=idle, hi=hwintr, si=softintr. So in my >example 1% in user space, 7.3% in system level, 3.3% in h/w intr, >and 15% in soft intr. Total of 26.6% utilized of which 25.6% is >in kernel. > > >I think that ''top'' shows system time spent on behalf of user processes. Dom0 has stuff going on in the kernel that has nothing to do with user processes. For instance, when you generate lan traffic to/from another domain, it causes the dom0 kernel to use a bunch of cpu time, and this may not show up using ''top''. That''s my guess about what is happening. Rob _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
top isn''t smart enough to realize that for a portion of the running time, a domain has been pre-empted out and is not running. top will then charge whatever was running at pre-emption time for the time that other domain running. Regards, Anthony Liguori Dave Thompson (davetho) wrote:> > > > >>-----Original Message----- >>From: Rob Gardner [mailto:rob.gardner@hp.com] >>Sent: Monday, December 12, 2005 1:23 PM >>To: Dave Thompson (davetho) >>Cc: xen-devel@lists.xensource.com >>Subject: Re: [Xen-devel] CPU Utilization >> >>Dave Thompson (davetho) wrote: >> >> >> >>>I have a question concerning the CPU utilization number provided by >>>the ''xm top'' command in xen 3.0. I''ve noticed that if I run a TCP >>>performance tool on domain 0 which simply receives and echos back >>>a TCP stream, the output of ''xm top'' during the test does not match >>>up with the output of ''top'' on domain 0 in terms of CPU utilization >>>(nor with the CPU utilization computed by the test program). >>> >>>... >>> >>>Any idea why these displays disagree? >>> >>> >>> >>> >>Top shows cpu usage for user processes in dom0, while xm top >>shows cpu usage for the domain, which includes cpu time spent >>in the dom0 kernel. >> >> > >Doesn''t top''s CPU summary line show both? I thought us=user space, >sy=system, ni=nice, id=idle, hi=hwintr, si=softintr. So in my >example 1% in user space, 7.3% in system level, 3.3% in h/w intr, >and 15% in soft intr. Total of 26.6% utilized of which 25.6% is >in kernel. > > > >>Perhaps the thing you''re running incurs lots of kernel cpu >>time, but not >>user time? >> >>Rob >> >> >> > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> -----Original Message----- > From: Anthony Liguori [mailto:aliguori@us.ibm.com] > Sent: Monday, December 12, 2005 5:01 PM > To: Dave Thompson (davetho) > Cc: Rob Gardner; xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] CPU Utilization > > top isn''t smart enough to realize that for a portion of the running > time, a domain has been pre-empted out and is not running. > > top will then charge whatever was running at pre-emption time for the > time that other domain running.But what else is running? In this case I only have dom0 configured, there is no domU. The only other possibility would be the hypervisor and I hope the hypervisor is not accounting for the other 30%.> Regards, > > Anthony Liguori >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dave Thompson (davetho) wrote:> > > > >>-----Original Message----- >>From: Anthony Liguori [mailto:aliguori@us.ibm.com] >>Sent: Monday, December 12, 2005 5:01 PM >>To: Dave Thompson (davetho) >>Cc: Rob Gardner; xen-devel@lists.xensource.com >>Subject: Re: [Xen-devel] CPU Utilization >> >>top isn''t smart enough to realize that for a portion of the running >>time, a domain has been pre-empted out and is not running. >> >>top will then charge whatever was running at pre-emption time for the >>time that other domain running. >> >> > >But what else is running? In this case I only have dom0 configured, >there is no domU. The only other possibility would be the hypervisor >and I hope the hypervisor is not accounting for the other 30%. > >If xend is started, you may have the software bridge running which can use as much as 10% cpu. Also, I don''t see soft ints in that top output. That could also be another ~7% cpu. Also xen is doing some work, receiving the real interrupts and generating virtual interrupts to dom0, so with all this, it is possible that you are using another 30% unseen in top. Best way to confirm this would be to use xenoprofile. -Andrew> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, 12 Dec 2005, Anthony Liguori wrote:> top isn''t smart enough to realize that for a portion of the running > time, a domain has been pre-empted out and is not running. > > top will then charge whatever was running at pre-emption time for the > time that other domain running.Actually, the kernel does this charging of time to the wrong entity. I am working on a patch to account the preempted time as "cpu steal" time (like s390 Linux does). Here is the (really simple) hypervisor bit of the patch. The kernel bits need some more work ;) Index: xen-3.0/xen/common/domain.c ==================================================================--- xen-3.0.orig/xen/common/domain.c +++ xen-3.0/xen/common/domain.c @@ -431,8 +431,25 @@ long do_vcpu_op(int cmd, int vcpuid, voi case VCPUOP_is_up: rc = !test_bit(_VCPUF_down, &v->vcpu_flags); break; + + case VCPUOP_cpu_time: + { + dom0_getvcpuinfo_t vi; + vi.online = !test_bit(_VCPUF_down, &v->vcpu_flags); + vi.blocked = test_bit(_VCPUF_blocked, &v->vcpu_flags); + vi.running = test_bit(_VCPUF_running, &v->vcpu_flags); + vi.cpu_time = v->cpu_time; + vi.cpu = v->processor; + vi.cpumap = v->cpumap; + rc = 0; + + if ( copy_to_user(arg, &vi, sizeof(dom0_getvcpuinfo_t)) ) + rc = -EFAULT; + break; + } } + return rc; } Index: xen-3.0/xen/include/public/vcpu.h ==================================================================--- xen-3.0.orig/xen/include/public/vcpu.h +++ xen-3.0/xen/include/public/vcpu.h @@ -51,6 +51,14 @@ /* Returns 1 if the given VCPU is up. */ #define VCPUOP_is_up 3 +/* + * Get information on how much CPU time this VCPU has used, etc... + * + * @extra_arg == pointer to an empty dom0_getvcpuinfo_t, the "OUT" variables + * of which filled in with scheduler info. + */ +#define VCPUOP_cpu_time 4 + #endif /* __XEN_PUBLIC_VCPU_H__ */ /* _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> -----Original Message----- > From: Andrew Theurer [mailto:habanero@us.ibm.com] > Sent: Monday, December 12, 2005 9:24 PM > To: Dave Thompson (davetho) > Cc: Anthony Liguori; xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] CPU Utilization > > > > >But what else is running? In this case I only have dom0 configured, > >there is no domU. The only other possibility would be the hypervisor > >and I hope the hypervisor is not accounting for the other 30%. > > > > If xend is started, you may have the software bridge running > which can use as much as 10% cpu.But I would think that the bridge activity should be showing up in the top CPU summary as well. It is running on domain 0 after all. I know one person suggested that kernel activity is not represented in the top CPU util output. But I don''t see how that can be right. If so, where else is that time accounted for? It seems to be all there (in the sy, hi, and si values).> Also, I don''t see soft ints in that top output. > That could also be another ~7% cpu.Soft interrupt time is accounted for in the si field (15%) of the summary. I believe that is where most (if not all) of the TCP processing is performed. Here is the top CPU summary display again: Cpu(s): 1.0% us, 7.3% sy, 0.0% ni, 73.3% id, 0.0% wa, 3.3% hi, 15.0% si> Also xen is doing some work, receiving the real interrupts > and generating virtual interrupts to dom0, so with all this, > it is possible that you are using another 30% unseen > in top.But aren''t the hypervisor calls actually still being accounted for by the domain since clock ticks are not lost but made up for in the timer_interrupt() function of arch/xen/i386/kiernel/time.c? The only issue is really when a domain is preempted by another domain by the xen scheduler and this is actually a problem in the other direction. The swapped out domain will still account for the time in whichever time bucket it was using when the domain was preempted (so the same time is accounted for by both domains). Basically the aggregated CPU time for all domains on a CPU could add greater than 100% because of this. If the domain is re-scheduled because of a SCHEDOP_block in the idle loop, the time will be properly accounted for as idle time. However, none of this really matters for my case since I am only running domain 0, there is no guest domain. I just want a good explanation why ''xm top'' is reporting 30% more CPU utilization than top in this case.> Best way to confirm this would be to use xenoprofile.Xenoprof is great for seeing which kernel functions are taking the majority of time but does it really help with CPU utilization? It counts (in the default case) unhalted clock cycles and in the xen idle loop the processor is halted (to save power) so the clock cycles are not accounted for. Is this right or am I missing something.> -AndrewThanks, Dave _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dave Thompson (davetho) wrote:>>-----Original Message----- >>From: Andrew Theurer [mailto:habanero@us.ibm.com] >>Sent: Monday, December 12, 2005 9:24 PM >>To: Dave Thompson (davetho) >>Cc: Anthony Liguori; xen-devel@lists.xensource.com >>Subject: Re: [Xen-devel] CPU Utilization >> >> >>>But what else is running? In this case I only have dom0 configured, >>>there is no domU. The only other possibility would be the hypervisor >>>and I hope the hypervisor is not accounting for the other 30%. >>> >>> >>If xend is started, you may have the software bridge running >>which can use as much as 10% cpu. >> > >But I would think that the bridge activity should be showing up >in the top CPU summary as well. It is running on domain 0 after all. >I know one person suggested that kernel activity is not represented >in the top CPU util output. But I don''t see how that can be right. >If so, where else is that time accounted for? It seems to be all >there (in the sy, hi, and si values). > > >>Also, I don''t see soft ints in that top output. >>That could also be another ~7% cpu. >> > >Soft interrupt time is accounted for in the si field (15%) of the >summary. I believe that is where most (if not all) of the TCP >processing is performed. Here is the top CPU summary display again: > >Cpu(s): 1.0% us, 7.3% sy, 0.0% ni, 73.3% id, 0.0% wa, 3.3% hi, >15.0% si > >Sorry, I overlooked the si.>>Also xen is doing some work, receiving the real interrupts >>and generating virtual interrupts to dom0, so with all this, >>it is possible that you are using another 30% unseen >>in top. >> > >But aren''t the hypervisor calls actually still being accounted for >by the domain since clock ticks are not lost but made up for in the >timer_interrupt() function of arch/xen/i386/kiernel/time.c? The >only issue is really when a domain is preempted by another domain >by the xen scheduler and this is actually a problem in the other >direction. The swapped out domain will still account for the >time in whichever time bucket it was using when the domain was >preempted (so the same time is accounted for by both domains). >Basically the aggregated CPU time for all domains on a CPU could >add greater than 100% because of this. If the domain is >re-scheduled because of a SCHEDOP_block in the idle loop, the time >will be properly accounted for as idle time. >I wonder if this is working under all situations. This problem seems familiar. Before the kernel accounted for si and hi properly, we had a very similar situation with this type of workload: lots of cpu time unaccounted for because the interrupt processing happend mostly when the system was idle, and the timer tick did not account for this properly. I wonder if we have a similar problem in xen/linux. If lost ticks are "queued up" but accounted for just one type of mode, then I think we could be way off in some sitations like this.> >However, none of this really matters for my case since I am >only running domain 0, there is no guest domain. I just want >a good explanation why ''xm top'' is reporting 30% more CPU utilization >than top in this case. > > >>Best way to confirm this would be to use xenoprofile. >> > >Xenoprof is great for seeing which kernel functions are taking >the majority of time but does it really help with CPU utilization? >It counts (in the default case) unhalted clock cycles and in the >xen idle loop the processor is halted (to save power) so the >clock cycles are not accounted for. Is this right or am I >missing something. >I guess I was hoping to find a smoking gun in xen :). The only other thing I think we could do is count the number of total samples we got over x seconds and compare this with the number of samples we would get in the same time period on a 100% busy system. We should then be able to figure out how much % time the cpu was halted. -Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel