Hi, I have been trying to run xenmon in its logging mode to collect statistics for offline analysis. However I notice that at an interval of 100 ms or so the overhead imposed by dom0 is quite high , with upto 40% of CPU utilization. Are there ways I can drastically reduce this overhead ? I read somwhere about event overhead, using event masks and python overhead, but am not really sure how to use this information to reduce the overhead. Any help is greatly appreciated ! thanks, Ashish _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Ashish,> for offline analysis. However I notice that at an interval of 100 ms or so > the overhead imposed by dom0 is quite high , with upto 40% of CPU > utilization.Are you sure that the overhead is due to xenmon? What CPU utilization do you see in Domain 0 for the same workload, but without xenmon?> Are there ways I can drastically reduce this overhead ? I read somwhere > about event overhead, using event masks and python overhead, but am not > really sure how to use this information to reduce the overhead.Exactly what metrics are you interested in monitoring? You might want to look at ''xentrace --help'' and ''xentrace -e''. Basically you want the minimal set of events that will give you the desired information. To see available events, look at /usr/include/xen/trace.h HTH, Diwaker -- Web/Blog/Gallery: http://floatingsun.net/blog _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thanks Diwaker for your response and the great tool you have developed ! The dom0 util is 2-3 % without xenmon. In the normal ncurses mode with 1 second interval, the overhead is almost negligible. But at 100ms logging mode its much more as I said. I am running a compute/communicate intensive app as one of the guests and trying to collect its stats at a sub-second level. I am mainly interested in monitoring the cpu, block, waiting times for now, and possibly the number if IOs and execs per second also at a later stage. Do you think cutting down on events collected can greatly reduce the overhead ? Also, is xentrace run as part of xenmon ? I am not absolutely clear where can I specify the event mask. I am also wondering if writing the xenmon part in C or hard coding my requirements in xenbaked (for my purpose) could greatly reduce the overhead ? Any suggestions would be very helpful ! thanks, Ashish On 5/29/07, Diwaker Gupta <diwaker.lists@gmail.com> wrote:> > Ashish, > > > for offline analysis. However I notice that at an interval of 100 ms or > so > > the overhead imposed by dom0 is quite high , with upto 40% of CPU > > utilization. > > Are you sure that the overhead is due to xenmon? What CPU utilization > do you see in Domain 0 for the same workload, but without xenmon? > > > Are there ways I can drastically reduce this overhead ? I read somwhere > > about event overhead, using event masks and python overhead, but am not > > really sure how to use this information to reduce the overhead. > > Exactly what metrics are you interested in monitoring? You might want > to look at ''xentrace --help'' and ''xentrace -e''. Basically you want the > minimal set of events that will give you the desired information. To > see available events, look at /usr/include/xen/trace.h > > HTH, > Diwaker > -- > Web/Blog/Gallery: http://floatingsun.net/blog >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > Ashish Gupta > Sent: 29 May 2007 20:54 > To: Xen-users@lists.xensource.com > Subject: [Xen-users] decreasing Xenmon overhead > > Hi, > > I have been trying to run xenmon in its logging mode to > collect statistics for offline analysis. However I notice > that at an interval of 100 ms or so the overhead imposed by > dom0 is quite high , with upto 40% of CPU utilization.Yes, simply don''t use 100ms inervals?> > Are there ways I can drastically reduce this overhead ? I > read somwhere about event overhead, using event masks and > python overhead, but am not really sure how to use this > information to reduce the overhead.Perhaps write a C-program rather than using Python to fetch/format the information? That shouls improve it, but I guess you''ll still see a noticable overhead if you sample the information every 100ms. I''m not sure it''s that meaningfull to run it that frequently, really. -- Mats> > Any help is greatly appreciated ! > > thanks, > Ashish > > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> > Are there ways I can drastically reduce this overhead ? I > > read somwhere about event overhead, using event masks and > > python overhead, but am not really sure how to use this > > information to reduce the overhead. > > Perhaps write a C-program rather than using Python to fetch/format the > information? That shouls improve it, but I guess you''ll still see a > noticable overhead if you sample the information every 100ms.xentrace (for instance) just logs out data to a file, and does the formatting offline using a python script. This is also a useful approach if you''re getting data out at a high bandwidth. Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: M.A. Williamson [mailto:maw48@hermes.cam.ac.uk] On > Behalf Of Mark Williamson > Sent: 30 May 2007 13:11 > To: xen-users@lists.xensource.com > Cc: Petersson, Mats; Ashish Gupta > Subject: Re: [Xen-users] decreasing Xenmon overhead > > > > Are there ways I can drastically reduce this overhead ? I > > > read somwhere about event overhead, using event masks and > > > python overhead, but am not really sure how to use this > > > information to reduce the overhead. > > > > Perhaps write a C-program rather than using Python to > fetch/format the > > information? That shouls improve it, but I guess you''ll still see a > > noticable overhead if you sample the information every 100ms. > > xentrace (for instance) just logs out data to a file, and > does the formatting > offline using a python script. This is also a useful > approach if you''re > getting data out at a high bandwidth.Very good point. Just "storing" the data in a trivial format is much less CPU-intensive than doing a lot of processing for each process step. And hopefully, when you''ve finished the testing/logging, you don''t care as much about how big the load is on the system. -- Mats> > Cheers, > Mark > > > -- > Dave: Just a question. What use is a unicyle with no seat? > And no pedals! > Mark: To answer a question with a question: What use is a skateboard? > Dave: Skateboards have wheels. > Mark: My wheel has a wheel! > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
These are some great points. As you said earlier, is 100 ms resolution meaningless ? Does it mean that the data emitted by xentrace is not accurate enough at that level ? So, I would have to do all the processing of xenbaked and xenmon in one module offline, over the xentrace data. Does this seem like a fair approach ? thanks!, Ashish On 5/30/07, Petersson, Mats <Mats.Petersson@amd.com> wrote:> > > > > -----Original Message----- > > From: M.A. Williamson [mailto:maw48@hermes.cam.ac.uk] On > > Behalf Of Mark Williamson > > Sent: 30 May 2007 13:11 > > To: xen-users@lists.xensource.com > > Cc: Petersson, Mats; Ashish Gupta > > Subject: Re: [Xen-users] decreasing Xenmon overhead > > > > > > Are there ways I can drastically reduce this overhead ? I > > > > read somwhere about event overhead, using event masks and > > > > python overhead, but am not really sure how to use this > > > > information to reduce the overhead. > > > > > > Perhaps write a C-program rather than using Python to > > fetch/format the > > > information? That shouls improve it, but I guess you''ll still see a > > > noticable overhead if you sample the information every 100ms. > > > > xentrace (for instance) just logs out data to a file, and > > does the formatting > > offline using a python script. This is also a useful > > approach if you''re > > getting data out at a high bandwidth. > > Very good point. Just "storing" the data in a trivial format is much > less CPU-intensive than doing a lot of processing for each process step. > And hopefully, when you''ve finished the testing/logging, you don''t care > as much about how big the load is on the system. > > -- > Mats > > > > Cheers, > > Mark > > > > > > -- > > Dave: Just a question. What use is a unicyle with no seat? > > And no pedals! > > Mark: To answer a question with a question: What use is a skateboard? > > Dave: Skateboards have wheels. > > Mark: My wheel has a wheel! > > > > > > > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >-- http://www.cs.northwestern.edu/~agupta _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users