Hi there,
I am getting very low throughput (around 0.29Mbps) while running netperf
benchmark for guest to guest communication on a single physical host. To analyse
where the time is spent in hypervisor I wanna use profiling. Please help me
choosing good profiler according to my requirments. Should it be better in my
case, Xentrace, gprof, xenoprof or Oprofile. Is it possible to use some vtune
version on FC11?
Regards,
Fasiha Ashraf
Yahoo! India has a new look. Take a sneak peek http://in.yahoo.com/trynew
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
I experienced the same problem with netperf and with hping also. In general every tool that sends packet at fixed rates seems to suffer this problem. If you use the flood option ( sends packets as fast as possible - for example ''ping -f'' or ''hping3 -f'') instead of a fixed rate the rate become much higher. What I discovered is this issue should be related to Xen''s signal handling/delivery. Netperf use setitimer() to send packets at fixed rate. In my experience on dom0/U, with no load, Xen cannot go over 250 signals per second. Because netperf use signal wakeup the function that send packets I think your problem can be related to this issue. If you want to be sure, count the average number of packet per seconds. It should be about 250 packets/s (not more). For more info you can see this: http://lists.xensource.com/archives/html/xen-devel/2009-09/msg01159.html Regards. Marco On Wed, Sep 30, 2009 at 8:07 AM, Fasiha Ashraf <feehapk@yahoo.co.in> wrote:> Hi there, > I am getting very low throughput (around 0.29Mbps) while running netperf > benchmark for guest to guest communication on a single physical host. To > analyse where the time is spent in hypervisor I wanna use profiling. Please > help me choosing good profiler according to my requirments. Should it be > better in my case, Xentrace, gprof, xenoprof or Oprofile. Is it possible to > use some vtune version on FC11? >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Sep 30, 2009 at 08:33:19AM +0200, Marco Tizzoni wrote:> I experienced the same problem with netperf and with hping also. In > general every tool that sends packet at fixed rates seems to suffer this > problem. If you use the flood option ( sends packets as fast as possible - > for example ''ping -f'' or ''hping3 -f'') instead of a fixed rate the rate > become much higher. What I discovered is this issue should be related to > Xen''s signal handling/delivery. > Netperf use setitimer() to send packets at fixed rate. In my experience on > dom0/U, with no load, Xen cannot go over 250 signals per second. Because > netperf use signal wakeup the function that send packets I think your > problem can be related to this issue. If you want to be sure, count the > average number of packet per seconds. It should be about 250 packets/s > (not more).250 per second.. sounds like kernel HZ value? -- Pasi> For more info you can see > this:[1]http://lists.xensource.com/archives/html/xen-devel/2009-09/msg01159.html > > Regards. > Marco > > On Wed, Sep 30, 2009 at 8:07 AM, Fasiha Ashraf <[2]feehapk@yahoo.co.in> > wrote: > > Hi there, > I am getting very low throughput (around 0.29Mbps) while running netperf > benchmark for guest to guest communication on a single physical host. To > analyse where the time is spent in hypervisor I wanna use profiling. > Please help me choosing good profiler according to my requirments. > Should it be better in my case, Xentrace, gprof, xenoprof or Oprofile. > Is it possible to use some vtune version on FC11? > > References > > Visible links > 1. http://lists.xensource.com/archives/html/xen-devel/2009-09/msg01159.html > 2. mailto:feehapk@yahoo.co.in> _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Sep 30, 2009 at 8:37 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:> On Wed, Sep 30, 2009 at 08:33:19AM +0200, Marco Tizzoni wrote: >> I experienced the same problem with netperf and with hping also. In >> general every tool that sends packet at fixed rates seems to suffer this >> problem. If you use the flood option ( sends packets as fast as possible - >> for example ''ping -f'' or ''hping3 -f'') instead of a fixed rate the rate >> become much higher. What I discovered is this issue should be related to >> Xen''s signal handling/delivery. >> Netperf use setitimer() to send packets at fixed rate. In my experience on >> dom0/U, with no load, Xen cannot go over 250 signals per second. Because >> netperf use signal wakeup the function that send packets I think your >> problem can be related to this issue. If you want to be sure, count the >> average number of packet per seconds. It should be about 250 packets/s >> (not more). > > > 250 per second.. sounds like kernel HZ value?Yes, infact I''m going to test it with 1000Khz. Stay tuned. :) m- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Sep 30, 2009 at 8:40 AM, Marco Tizzoni <marco.tizzoni@gmail.com> wrote:> On Wed, Sep 30, 2009 at 8:37 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote: >> 250 per second.. sounds like kernel HZ value? > > Yes, infact I''m going to test it with 1000Khz. Stay tuned.Building dom0 kernel with 1000hz the rate reach 1k. Ok, one point. Then? Why this behaviour occurs only on xenolinux? m- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Enabling debug on kernel I realized the problem is related to timer accuracy. ------------------------------------------------------------------------ hal9k-dom0 ~ # cat /proc/timer_list Timer List Version: v0.3 HRTIMER_MAX_CLOCK_BASES: 2 now at 784689453083 nsecs cpu: 0 clock 0: .index: 0 .resolution: 999848 nsecs [........] hal9k-dom0 ~ # cat /sys/devices/system/clocksource/clocksource0/available_clocksource xen jiffies hal9k-dom0 ~ # cat /sys/devices/system/clocksource/clocksource0/current_clocksource xen ------------------------------------------------------------------------ Is there a way to make available a better timer (hpet,acpi....)? Also, on another xen box (rhel 5.2), I''ve a ''pit'' clocksource in available_clocksource. After: # sysctl -w xen.independent_wallclock=1 and ------------------------------------------------------------------------------------------------------------------------------------ # echo pit > /sys/devices/system/clocksource/clocksource0/current_clocksource [root@station1 init.d]# cat /sys/devices/system/clocksource/clocksource0/current_clocksource jiffies ------------------------------------------------------------------------------------------------------------------------------------ the clocksource is unmodified. m- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thanks alot for the useful information. I will give it a try what is suggested. Kindly tell me one more thing. I dont found "xenbr0" interface while running "ifconfig -a" command in "dom0". It doesn''t come with xen3.4.2 and FC11 pv os kernel 2.6.30-rc6-tip or there''s some problem here on my side? Regards, Fasiha Ashraf --- On Wed, 30/9/09, Marco Tizzoni <marco.tizzoni@gmail.com> wrote: From: Marco Tizzoni <marco.tizzoni@gmail.com> Subject: Re: [Xen-devel] Re: [Xen-users] About profiling xen To: "Pasi Kärkkäinen" <pasik@iki.fi> Cc: "Fajar A." <fajar@fajar.net>, "Fasiha Ashraf" <feehapk@yahoo.co.in>, "xen" <xen-devel@lists.xensource.com>, xen-users@lists.xensource.com Date: Wednesday, 30 September, 2009, 12:45 PM On Wed, Sep 30, 2009 at 8:40 AM, Marco Tizzoni <marco.tizzoni@gmail.com> wrote:> On Wed, Sep 30, 2009 at 8:37 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote: >> 250 per second.. sounds like kernel HZ value? > > Yes, infact I''m going to test it with 1000Khz. Stay tuned.Building dom0 kernel with 1000hz the rate reach 1k. Ok, one point. Then? Why this behaviour occurs only on xenolinux? m- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel Try the new Yahoo! India Homepage. Click here. http://in.yahoo.com/trynew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Sep 30, 2009 at 02:46:46PM +0530, Fasiha Ashraf wrote:> Thanks alot for the useful information. I will give it a try what is > suggested. > Kindly tell me one more thing. I dont found "xenbr0" interface while > running "ifconfig -a" command in "dom0". It doesn''t come with xen3.4.2 and > FC11 pv os kernel 2.6.30-rc6-tip or there''s some problem here on my side? >Btw you should use 2.6.31 pv_ops dom0 kernel (from xen/master) branch, since that''s the only "supported" and maintained branch now. -- Pasi> Regards, > Fasiha Ashraf > > --- On Wed, 30/9/09, Marco Tizzoni <marco.tizzoni@gmail.com> wrote: > > From: Marco Tizzoni <marco.tizzoni@gmail.com> > Subject: Re: [Xen-devel] Re: [Xen-users] About profiling xen > To: "Pasi Kärkkäinen" <pasik@iki.fi> > Cc: "Fajar A." <fajar@fajar.net>, "Fasiha Ashraf" <feehapk@yahoo.co.in>, > "xen" <xen-devel@lists.xensource.com>, xen-users@lists.xensource.com > Date: Wednesday, 30 September, 2009, 12:45 PM > > On Wed, Sep 30, 2009 at 8:40 AM, Marco Tizzoni > <[1]marco.tizzoni@gmail.com> wrote: > > On Wed, Sep 30, 2009 at 8:37 AM, Pasi Kärkkäinen <[2]pasik@iki.fi> > wrote: > >> 250 per second.. sounds like kernel HZ value? > > > > Yes, infact I''m going to test it with 1000Khz. Stay tuned. > > Building dom0 kernel with 1000hz the rate reach 1k. Ok, one point. Then? > Why this behaviour occurs only on xenolinux? > > m- > > _______________________________________________ > Xen-devel mailing list > [3]Xen-devel@lists.xensource.com > [4]http://lists.xensource.com/xen-devel > > -------------------------------------------------------------------------- > > Now, send attachments up to 25MB with Yahoo! India Mail. [5]Learn how. > > References > > Visible links > 1. file:///mc/compose?to=marco.tizzoni@gmail.com > 2. file:///mc/compose?to=pasik@iki.fi > 3. file:///mc/compose?to=Xen-devel@lists.xensource.com > 4. http://lists.xensource.com/xen-devel > 5. http://in.rd.yahoo.com/tagline_galaxy_2/*http:/in.overview.mail.yahoo.com/photos_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Is there a way to make available a better timer (hpet,acpi....)? > > Also, on another xen box (rhel 5.2), I''ve a ''pit'' clocksource in > available_clocksource. > After: > # sysctl -w xen.independent_wallclock=1 > > and > -------------------------------------------------------------- > ---------------------------------------------------------------------- > # echo pit > > /sys/devices/system/clocksource/clocksource0/current_clocksource > [root@station1 init.d]# cat > /sys/devices/system/clocksource/clocksource0/current_clocksource > jiffies > -------------------------------------------------------------- > ---------------------------------------------------------------------- > > the clocksource is unmodified. >Because of save/restore and migrate, no hardware timer is suitable for a pv domain. The code that is handling the clock is the paravirtualized clock code (aka pvclock). Nobody ever changed the sysfs reporting mechanism in pv versions of released Red Hat kernels, so clocksource gets reported as "jiffies", and even though "pit" appears to be available, it is not (since it wouldn''t work). I suspect this is also the case for other distros'' pv kernels. Dan _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, Sep 30, 2009 at 4:21 PM, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:> Because of save/restore and migrate, no hardware timer is > suitable for a pv domain. The code that is handling the > clock is the paravirtualized clock code (aka pvclock). > Nobody ever changed the sysfs reporting mechanism in > pv versions of released Red Hat kernels, so clocksource > gets reported as "jiffies", and even though "pit" > appears to be available, it is not (since it wouldn''t > work). I suspect this is also the case for other > distros'' pv kernels.How can I get a different, and I hope better, clocksource? In time.c I found acpi and hpet are available. m- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> On Wed, Sep 30, 2009 at 4:21 PM, Dan Magenheimer > <dan.magenheimer@oracle.com> wrote: > > > Because of save/restore and migrate, no hardware timer is > > suitable for a pv domain. The code that is handling the > > clock is the paravirtualized clock code (aka pvclock). > > Nobody ever changed the sysfs reporting mechanism in > > pv versions of released Red Hat kernels, so clocksource > > gets reported as "jiffies", and even though "pit" > > appears to be available, it is not (since it wouldn''t > > work). I suspect this is also the case for other > > distros'' pv kernels. > > How can I get a different, and I hope better, clocksource? In time.c I > found acpi and hpet are available.You can''t. Paravirtualization disables all other clocksources. You can modify the source to change HZ and rebuild (which you apparently already did), or you can use an HVM instead (but that will change many things, so you will be comparing apples and oranges). I''m no networking expert, but if your packets are only being processed once per HZ, that seems like a completely different problem. Even if you are able to get a hardware high resolution timer, the resolution has nothing to do with the clock interrupt frequency (HZ). _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Sep 30, 2009 at 6:04 PM, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:>> How can I get a different, and I hope better, clocksource? In time.c I >> found acpi and hpet are available. > > > I''m no networking expert, but if your packets are > only being processed once per HZ, that seems like > a completely different problem. Even if you are > able to get a hardware high resolution timer, the > resolution has nothing to do with the clock > interrupt frequency (HZ).Netperf use setitimer() to send packets at regular intervals to test network speed. If the timer is not finer enough the signal is not delivered on time and becomes impossibile to reach the requested rate, i.e. if timer is 250Hz you can get at most 250 alarm/s.>From my point of view this issue should be documented and as well asthe clocksource option. I cactched some information googling but it''s not clearly explained. m- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Sep-30 23:19 UTC
Re: [Xen-devel] Re: [Xen-users] About profiling xen
On 09/30/09 00:45, Marco Tizzoni wrote:> On Wed, Sep 30, 2009 at 8:40 AM, Marco Tizzoni <marco.tizzoni@gmail.com> wrote: > >> On Wed, Sep 30, 2009 at 8:37 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote: >> >>> 250 per second.. sounds like kernel HZ value? >>> >> Yes, infact I''m going to test it with 1000Khz. Stay tuned. >> > Building dom0 kernel with 1000hz the rate reach 1k. Ok, one point. Then? > Why this behaviour occurs only on xenolinux? >Which kernel are you using? Do you have: CONFIG_NO_HZ=y CONFIG_HIGH_RES_TIMERS=y configured? J _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jeremy Fitzhardinge
2009-Sep-30 23:23 UTC
Re: [Xen-devel] Re: [Xen-users] About profiling xen
On 09/30/09 02:06, Marco Tizzoni wrote:> Enabling debug on kernel I realized the problem is related to timer accuracy. > ------------------------------------------------------------------------ > hal9k-dom0 ~ # cat /proc/timer_list > Timer List Version: v0.3 > HRTIMER_MAX_CLOCK_BASES: 2 > now at 784689453083 nsecs > > cpu: 0 > clock 0: > .index: 0 > .resolution: 999848 nsecs > [........] > hal9k-dom0 ~ # cat > /sys/devices/system/clocksource/clocksource0/available_clocksource > xen jiffies > hal9k-dom0 ~ # cat > /sys/devices/system/clocksource/clocksource0/current_clocksource > xen > ------------------------------------------------------------------------ > > Is there a way to make available a better timer (hpet,acpi....)? >Your clocksource is "xen", which is the best possible for a PV xen guest. However, a clocksource is for measuring elapsed time, not triggering timers. Unfortunately there doesn''t seem to be a /sys file to show the current clockevent source in use, but if you have "xen" clocksource it''s almost certainly the xen clockevent. However, this is only relevent if you have CONFIG_NO_HZ and CONFIG_HIGHRES_TIMERS enabled. If you don''t then all timers will be quantized to HZ (and your guests will burn lots CPU being needlessly interrupted at HZ). J _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thanks, Ok! I''ll give it a try. But let me tell you that the compile n build was quite ok. Just the problem occurs when it comes to Guest communication on single physical host. Regards, Fasiha Ashraf --- On Wed, 30/9/09, Pasi Kärkkäinen <pasik@iki.fi> wrote: From: Pasi Kärkkäinen <pasik@iki.fi> Subject: Re: [Xen-devel] Re: [Xen-users] About profiling xen To: "Fasiha Ashraf" <feehapk@yahoo.co.in> Cc: "Marco Tizzoni" <marco.tizzoni@gmail.com>, "Fajar A." <fajar@fajar.net>, "xen" <xen-devel@lists.xensource.com>, xen-users@lists.xensource.com Date: Wednesday, 30 September, 2009, 2:23 PM On Wed, Sep 30, 2009 at 02:46:46PM +0530, Fasiha Ashraf wrote:> Thanks alot for the useful information. I will give it a try what is > suggested. > Kindly tell me one more thing. I dont found "xenbr0" interface while > running "ifconfig -a" command in "dom0". It doesn''t come with xen3.4.2 and > FC11 pv os kernel 2.6.30-rc6-tip or there''s some problem here on my side? >Btw you should use 2.6.31 pv_ops dom0 kernel (from xen/master) branch, since that''s the only "supported" and maintained branch now. -- Pasi> Regards, > Fasiha Ashraf > > --- On Wed, 30/9/09, Marco Tizzoni <marco.tizzoni@gmail.com> wrote: > > From: Marco Tizzoni <marco.tizzoni@gmail.com> > Subject: Re: [Xen-devel] Re: [Xen-users] About profiling xen > To: "Pasi Kärkkäinen" <pasik@iki.fi> > Cc: "Fajar A." <fajar@fajar.net>, "Fasiha Ashraf" <feehapk@yahoo.co.in>, > "xen" <xen-devel@lists.xensource.com>, xen-users@lists.xensource.com > Date: Wednesday, 30 September, 2009, 12:45 PM > > On Wed, Sep 30, 2009 at 8:40 AM, Marco Tizzoni > <[1]marco.tizzoni@gmail.com> wrote: > > On Wed, Sep 30, 2009 at 8:37 AM, Pasi Kärkkäinen <[2]pasik@iki.fi> > wrote: > >> 250 per second.. sounds like kernel HZ value? > > > > Yes, infact I''m going to test it with 1000Khz. Stay tuned. > > Building dom0 kernel with 1000hz the rate reach 1k. Ok, one point. Then? > Why this behaviour occurs only on xenolinux? > > m- > > _______________________________________________ > Xen-devel mailing list > [3]Xen-devel@lists.xensource.com > [4]http://lists.xensource.com/xen-devel > > -------------------------------------------------------------------------- > > Now, send attachments up to 25MB with Yahoo! India Mail. [5]Learn how. > > References > > Visible links > 1. file:///mc/compose?to=marco.tizzoni@gmail.com > 2. file:///mc/compose?to=pasik@iki.fi > 3. file:///mc/compose?to=Xen-devel@lists.xensource.com > 4. http://lists.xensource.com/xen-devel > 5. http://in.rd.yahoo.com/tagline_galaxy_2/*http:/in.overview.mail.yahoo.com/photosYahoo! India has a new look. Take a sneak peek http://in.yahoo.com/trynew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Oct 1, 2009 at 1:23 AM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:> Your clocksource is "xen", which is the best possible for a PV xen guest. > > However, a clocksource is for measuring elapsed time, not triggering > timers.Why? How would you solve the problem of raising events at a fixed rate?> Unfortunately there doesn''t seem to be a /sys file to show the > current clockevent source in use, but if you have "xen" clocksource it''s > almost certainly the xen clockevent. > > However, this is only relevent if you have CONFIG_NO_HZ and > CONFIG_HIGHRES_TIMERS enabled.I''ve tried both set/unset but nothing change. m- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
While compiling Xen3.4 with linux kernel 2.6.31.1 At the end of build it
gives me a warning "WARNING: Package ''fig2dev'' is
required= to
build Xen documentation" What does that mean? Is that of no harm or it
will effect the performance and working of hypervisor in future?
I follow these steps for compilation
hg clone
http://xenbits.xensource.com/xen-unstable.hg
hg clone
http://xenbits.xensource.com/linux-2.6.18-xen.hg
cd xen-unstable.hg
make world
make install
this is how my /boot contains
-rw-r--r-- 1 root root 68783 2009-10-01 13:45 config-2.6.31.1
-rw-r--r-- 1 root root 1882027 2009-10-01 13:45
System.map-2.6.31.1
-rw-r--r-- 1 root root 4437728 2009-10-01 13:45 vmlinuz-2.6.31.1
lrwxrwxrwx 1 root root 19 2009-10-01 13:44 xen-3.5.gz ->
xen-3.5-unstable.gz
-rw-r--r-- 1 root root 517162 2009-10-01 13:44
xen-3.5-unstable.gz
lrwxrwxrwx 1 root root 19 2009-10-01 13:44 xen-3.gz ->
xen-3.5-unstable.gz
lrwxrwxrwx 1 root root 19 2009-10-01 13:44 xen.gz ->
xen-3.5-unstable.gz
-rw-r--r-- 1 root root 7370192 2009-10-01 13:44
xen-syms-3.5-unstable
there is no entry in grub.conf should I add it myself?
Regards
Fasiha Ashraf
--- On Wed, 30/9/09, Pasi Kärkkäinen <pasik@iki.fi> wrote:
From: Pasi Kärkkäinen <pasik@iki.fi>
Subject: Re: [Xen-devel] Re: [Xen-users] About profiling xen
To: "Fasiha Ashraf" <feehapk@yahoo.co.in>
Cc: "Marco Tizzoni" <marco.tizzoni@gmail.com>, "Fajar
A." <fajar@fajar.net>, "xen"
<xen-devel@lists.xensource.com>, xen-users@lists.xensource.com
Date: Wednesday, 30 September, 2009, 2:23 PM
On Wed, Sep 30, 2009 at 02:46:46PM +0530, Fasiha Ashraf
wrote:> Thanks alot for the useful information. I will give it a try what is
> suggested.
> Kindly tell me one more thing. I dont found "xenbr0" interface
while
> running "ifconfig -a" command in "dom0". It
doesn''t come with xen3.4.2 and
> FC11 pv os kernel 2.6.30-rc6-tip or there''s some problem here
on my side?
>
Btw you should use 2.6.31 pv_ops dom0 kernel (from xen/master) branch,
since that''s the only "supported" and maintained branch now.
-- Pasi
> Regards,
> Fasiha Ashraf
>
> --- On Wed, 30/9/09, Marco Tizzoni <marco.tizzoni@gmail.com>
wrote:
>
> From: Marco Tizzoni <marco.tizzoni@gmail.com>
> Subject: Re: [Xen-devel] Re: [Xen-users] About profiling xen
> To: "Pasi Kärkkäinen" <pasik@iki.fi>
> Cc: "Fajar A." <fajar@fajar.net>, "Fasiha
Ashraf" <feehapk@yahoo.co.in>,
> "xen" <xen-devel@lists.xensource.com>,
xen-users@lists.xensource.com
> Date: Wednesday, 30 September, 2009, 12:45 PM
>
> On Wed, Sep 30, 2009 at 8:40 AM, Marco Tizzoni
> <[1]marco.tizzoni@gmail.com> wrote:
> > On Wed, Sep 30, 2009 at 8:37 AM, Pasi Kärkkäinen
<[2]pasik@iki.fi>
> wrote:
> >> 250 per second.. sounds like kernel HZ value?
> >
> > Yes, infact I''m going to test it with 1000Khz. Stay
tuned.
>
> Building dom0 kernel with 1000hz the rate reach 1k. Ok, one point.
Then?
> Why this behaviour occurs only on xenolinux?
>
> m-
>
> _______________________________________________
> Xen-devel mailing list
> [3]Xen-devel@lists.xensource.com
> [4]http://lists.xensource.com/xen-devel
>
>
--------------------------------------------------------------------------
>
> Now, send attachments up to 25MB with Yahoo! India Mail. [5]Learn how.
>
> References
>
> Visible links
> 1. file:///mc/compose?to=marco.tizzoni@gmail.com
> 2. file:///mc/compose?to=pasik@iki.fi
> 3. file:///mc/compose?to=Xen-devel@lists.xensource.com
> 4. http://lists.xensource.com/xen-devel
> 5.
http://in.rd.yahoo.com/tagline_galaxy_2/*http:/in.overview.mail.yahoo.com/photos
Yahoo! India has a new look. Take a sneak peek http://in.yahoo.com/trynew
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Jeremy Fitzhardinge
2009-Oct-01 20:54 UTC
Re: [Xen-devel] Re: [Xen-users] About profiling xen
On 10/01/09 00:35, Marco Tizzoni wrote:> On Thu, Oct 1, 2009 at 1:23 AM, Jeremy Fitzhardinge <jeremy@goop.org> wrote: > >> Your clocksource is "xen", which is the best possible for a PV xen guest. >> >> However, a clocksource is for measuring elapsed time, not triggering >> timers. >> > Why? How would you solve the problem of raising events at a fixed rate? >I''m not sure I follow you. In the kernel, the clock*source* subsystem is for measuring time: its used to implement gettimeofday, as well as internal time accounting. It isn''t directly related to time events. The clock*event* mechanism is all about setting up timers to raise events. When running paravirtualized, we use Xen-specific versions of both.>> Unfortunately there doesn''t seem to be a /sys file to show the >> current clockevent source in use, but if you have "xen" clocksource it''s >> almost certainly the xen clockevent. >> >> However, this is only relevent if you have CONFIG_NO_HZ and >> CONFIG_HIGHRES_TIMERS enabled. >> > I''ve tried both set/unset but nothing change. >Hm. Its best to leave both enabled either way. Try the attached program to get some info about how well timers are working. Compile with: $ gcc -o testtimer testtimer.c -O -lrt And test various results: $ ./testtimer .1 > xen-0.1.out $ ./testtimer .01 > xen-0.01.out $ ./testtimer .001 > xen-0.001.out You can plot the results with: $ gnuplot gnuplot> plot "xen-0.001.out" with lines The plot is delta from the expected time, so the ideal is that they all be 0. On my system in dom0 I see about 10% overhead at .001 and .0005 sec, so the timers aren''t being quantized to 100/250Hz. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Oct 1, 2009 at 10:54 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:> On 10/01/09 00:35, Marco Tizzoni wrote: >> On Thu, Oct 1, 2009 at 1:23 AM, Jeremy Fitzhardinge <jeremy@goop.org> wrote: >> >>> Your clocksource is "xen", which is the best possible for a PV xen guest. >>> >>> However, a clocksource is for measuring elapsed time, not triggering >>> timers. >>> >> Why? How would you solve the problem of raising events at a fixed rate? >> > > I''m not sure I follow you. In the kernel, the clock*source* subsystem > is for measuring time: its used to implement gettimeofday, as well as > internal time accounting. It isn''t directly related to time events.Ok.> The clock*event* mechanism is all about setting up timers to raise events. > > When running paravirtualized, we use Xen-specific versions of both.Ok. My problem is granularity. In xen I can''t raise events more frequently than 1000/s (with timer set to 1000hz). Net test softwares such as netperf or hping use setitimer to send packet at a fixed rate. On a non-xen box they works fine even with 100k packets/sec (i.e. 100k alarms/sec). This can be problematic for streaming software and more in general for soft real-time applications.>>> Unfortunately there doesn''t seem to be a /sys file to show the >>> current clockevent source in use, but if you have "xen" clocksource it''s >>> almost certainly the xen clockevent. >>> >>> However, this is only relevent if you have CONFIG_NO_HZ and >>> CONFIG_HIGHRES_TIMERS enabled. >>> >> I''ve tried both set/unset but nothing change. >> > > Hm. Its best to leave both enabled either way. > > Try the attached program to get some info about how well timers are > working. Compile with:Thanks for your test, I tried and timers work as expected. m- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jeremy Fitzhardinge
2009-Oct-01 21:16 UTC
Re: [Xen-devel] Re: [Xen-users] About profiling xen
On 10/01/09 14:05, Marco Tizzoni wrote:>> The clock*event* mechanism is all about setting up timers to raise events. >> >> When running paravirtualized, we use Xen-specific versions of both. >> > Ok. My problem is granularity. In xen I can''t raise events more > frequently than 1000/s (with timer set to 1000hz). > Net test softwares such as netperf or hping use setitimer to send > packet at a fixed rate. On a non-xen box they works fine even with > 100k packets/sec (i.e. 100k alarms/sec). This can be problematic for > streaming software and more in general for soft real-time > applications. > [...] > Thanks for your test, I tried and timers work as expected.OK, good, that takes timers out of the equation. I guess the problem is the rate at which Xen will context switch between two domains. Hm. Does anything change if you pin your domains to different cpus? J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Oct 1, 2009 at 11:16 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:> On 10/01/09 14:05, Marco Tizzoni wrote: >>> The clock*event* mechanism is all about setting up timers to raise events. >>> >>> When running paravirtualized, we use Xen-specific versions of both. >>> >> Ok. My problem is granularity. In xen I can''t raise events more >> frequently than 1000/s (with timer set to 1000hz). >> Net test softwares such as netperf or hping use setitimer to send >> packet at a fixed rate. On a non-xen box they works fine even with >> 100k packets/sec (i.e. 100k alarms/sec). This can be problematic for >> streaming software and more in general for soft real-time >> applications. >> [...] >> Thanks for your test, I tried and timers work as expected. > > OK, good, that takes timers out of the equation. > > I guess the problem is the rate at which Xen will context switch between > two domains. Hm. > > Does anything change if you pin your domains to different cpus?mmmh, no. I think without a finer timer the problem could not be solved. For my test application, no problem, I''ll use a while() loop with counters and sched_yeld(), but, for real application, it could be a serious issue to deal with. m- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Oct-01 21:40 UTC
Re: [Xen-devel] Re: [Xen-users] About profiling xen
On 10/01/09 14:26, Marco Tizzoni wrote:>> OK, good, that takes timers out of the equation. >> >> I guess the problem is the rate at which Xen will context switch between >> two domains. Hm. >> >> Does anything change if you pin your domains to different cpus? >> > mmmh, no. I think without a finer timer the problem could not be solved. >Why''s that? You said that the timers "work as expected", which I read to mean that they will generate sub-millisecond events. Did I misunderstand you?> For my test application, no problem, I''ll use a while() loop with > counters and sched_yeld(), > but, for real application, it could be a serious issue to deal with. >I don''t see why that would be necessary if timers are working properly. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Oct 1, 2009 at 11:40 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:> On 10/01/09 14:26, Marco Tizzoni wrote: >>> OK, good, that takes timers out of the equation. >>> >>> I guess the problem is the rate at which Xen will context switch between >>> two domains. Hm. >>> >>> Does anything change if you pin your domains to different cpus? >>> >> mmmh, no. I think without a finer timer the problem could not be solved. >> > > Why''s that? You said that the timers "work as expected", which I read > to mean that they will generate sub-millisecond events. Did I > misunderstand you?Sorry, I was not clear. It works as I expect, i.e going not over 1000 hz (my dom0 kernel has been compiled with 1000hz). Following a couple of run of your testtimer: hal9k-dom0 ~ # ./testtimer .000001 |head -n10 1000 iterations at 0.000001 sec 0.000662 0.000946 0.000956 0.000960 0.000962 0.000956 0.000964 0.000963 0.000964 0.000964 hal9k-dom0 ~ # ./testtimer .00001 |head -n10 1000 iterations at 0.000010 sec 0.000407 0.000942 0.000941 0.000954 0.000956 0.000954 0.000955 0.000932 0.000955 0.000955 Hoping this clarify. m- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jeremy Fitzhardinge
2009-Oct-01 22:01 UTC
Re: [Xen-devel] Re: [Xen-users] About profiling xen
On 10/01/09 14:55, Marco Tizzoni wrote:> Sorry, I was not clear. It works as I expect, i.e going not over 1000 > hz (my dom0 kernel has been compiled with 1000hz). >OK, that''s definitely *not* expected. If you''re running with CONFIG_NO_HZ and HIGH_RES_TIMERS then timer resolution should have nothing to do with your HZ configuration.> Following a couple of run of your testtimer: > > hal9k-dom0 ~ # ./testtimer .000001 |head -n10 >I think expecting a timer to work at 1MHz is unreasonable.> 1000 iterations at 0.000001 sec > 0.000662 > 0.000946 > 0.000956 > 0.000960 > 0.000962 > 0.000956 > 0.000964 > 0.000963 > 0.000964 > 0.000964 > > hal9k-dom0 ~ # ./testtimer .00001 |head -n10 >100kHz is also very demanding. How does it work with 2kHz? 10kHz? J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Oct 2, 2009 at 12:01 AM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:> On 10/01/09 14:55, Marco Tizzoni wrote: >> Sorry, I was not clear. It works as I expect, i.e going not over 1000 >> hz (my dom0 kernel has been compiled with 1000hz). >> > > OK, that''s definitely *not* expected. If you''re running with > CONFIG_NO_HZ and HIGH_RES_TIMERS then timer resolution should have > nothing to do with your HZ configuration.> 100kHz is also very demanding. > How does it work with 2kHz? 10kHz?hal9k-dom0 ~ # ./testtimer .0001 |head -n10 1000 iterations at 0.000100 sec 0.000051 0.000885 0.000890 0.000892 0.000891 0.000892 0.000893 0.000893 0.000879 0.000893 hal9k-dom0 ~ # ./testtimer .002 |head -n10 1000 iterations at 0.002000 sec 0.000744 0.000978 0.000990 0.000990 0.000990 0.000991 0.000992 0.000991 0.000992 0.000990 m- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi, in my case guest to internet speed is ok. wget between guests shows 35K/s to the max.and the throughput in this case is 35.1K/s eta 4h 3m. I case of guest to host its around 147K/s As you said I think your problem can be related to this issue. If you want to be sure, count the average number of packet per seconds. It should be about 250 packets/s (not more). Packets per sec from guest to guest is 2packects per sec.>From guest to host is 1packet per sec >From Host to guest is 2packets per secRegards, Fasiha Ashraf --- On Wed, 30/9/09, Marco Tizzoni <marco.tizzoni@gmail.com> wrote: From: Marco Tizzoni <marco.tizzoni@gmail.com> Subject: [Xen-devel] Re: [Xen-users] About profiling xen To: "Fasiha Ashraf" <feehapk@yahoo.co.in> Cc: "Fajar A." <fajar@fajar.net>, "xen" <xen-devel@lists.xensource.com>, xen-users@lists.xensource.com Date: Wednesday, 30 September, 2009, 11:33 AM I experienced the same problem with netperf and with hping also. In general every tool that sends packet at fixed rates seems to suffer this problem. If you use the flood option ( sends packets as fast as possible - for example ''ping -f'' or ''hping3 -f'') instead of a fixed rate the rate become much higher. What I discovered is this issue should be related to Xen''s signal handling/delivery. Netperf use setitimer() to send packets at fixed rate. In my experience on dom0/U, with no load, Xen cannot go over 250 signals per second. Because netperf use signal wakeup the function that send packets I think your problem can be related to this issue. If you want to be sure, count the average number of packet per seconds. It should be about 250 packets/s (not more). For more info you can see this:http://lists.xensource.com/archives/html/xen-devel/2009-09/msg01159.html Regards. Marco On Wed, Sep 30, 2009 at 8:07 AM, Fasiha Ashraf <feehapk@yahoo.co.in> wrote: Hi there, I am getting very low throughput (around 0.29Mbps) while running netperf benchmark for guest to guest communication on a single physical host. To analyse where the time is spent in hypervisor I wanna use profiling. Please help me choosing good profiler according to my requirments. Should it be better in my case, Xentrace, gprof, xenoprof or Oprofile. Is it possible to use some vtune version on FC11? -----Inline Attachment Follows----- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel Keep up with people you care about with Yahoo! India Mail. Learn how. Now, send attachments up to 25MB with Yahoo! India Mail. Learn how. http://in.overview.mail.yahoo.com/photos _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Is this the issue that is affecting guest to guest throughput? Since I have timer frequency 1000Hz and high resolution timer both enabled in my dom0. Also i am using cup cores pinned for each VM an Dom0. Please guide me some more about getting increased throughput between guests. Could it be an ebtable filter issue? Regards, Fasiha Ashraf --- On Fri, 2/10/09, Marco Tizzoni <marco.tizzoni@gmail.com> wrote: From: Marco Tizzoni <marco.tizzoni@gmail.com> Subject: Re: [Xen-devel] Re: [Xen-users] About profiling xen To: "Jeremy Fitzhardinge" <jeremy@goop.org> Cc: "Fajar A." <fajar@fajar.net>, "Fasiha Ashraf" <feehapk@yahoo.co.in>, "xen" <xen-devel@lists.xensource.com>, xen-users@lists.xensource.com Date: Friday, 2 October, 2009, 3:20 AM On Fri, Oct 2, 2009 at 12:01 AM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:> On 10/01/09 14:55, Marco Tizzoni wrote: >> Sorry, I was not clear. It works as I expect, i.e going not over 1000 >> hz (my dom0 kernel has been compiled with 1000hz). >> > > OK, that''s definitely *not* expected. If you''re running with > CONFIG_NO_HZ and HIGH_RES_TIMERS then timer resolution should have > nothing to do with your HZ configuration.> 100kHz is also very demanding. > How does it work with 2kHz? 10kHz?hal9k-dom0 ~ # ./testtimer .0001 |head -n10 1000 iterations at 0.000100 sec 0.000051 0.000885 0.000890 0.000892 0.000891 0.000892 0.000893 0.000893 0.000879 0.000893 hal9k-dom0 ~ # ./testtimer .002 |head -n10 1000 iterations at 0.002000 sec 0.000744 0.000978 0.000990 0.000990 0.000990 0.000991 0.000992 0.000991 0.000992 0.000990 m- From cricket scores to your friends. Try the Yahoo! India Homepage! http://in.yahoo.com/trynew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Oct-05 20:53 UTC
Re: [Xen-devel] Re: [Xen-users] About profiling xen
On 10/05/09 04:38, Fasiha Ashraf wrote:> Is this the issue that is affecting guest to guest throughput? Since I > have timer frequency 1000Hz and high resolution timer both enabled in > my dom0. Also i am using cup cores pinned for each VM an Dom0. Please > guide me some more about getting increased throughput between guests. >I haven''t specifically looked at the problem of domain<->domain performance. The quantization of time at 1kHz is a concern, but its not clear to me whether it comes from Linux, Xen or some interaction between them. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Which IDE is best suitable for inspecting xen code? Netbeans or eclipse can be used? Regards, Fasiha Ashraf --- On Tue, 6/10/09, Jeremy Fitzhardinge <jeremy@goop.org> wrote: From: Jeremy Fitzhardinge <jeremy@goop.org> Subject: Re: [Xen-devel] Re: [Xen-users] About profiling xen To: "Fasiha Ashraf" <feehapk@yahoo.co.in> Cc: "Marco Tizzoni" <marco.tizzoni@gmail.com>, "Fajar A." <fajar@fajar.net>, "xen" <xen-devel@lists.xensource.com>, xen-users@lists.xensource.com Date: Tuesday, 6 October, 2009, 1:53 AM On 10/05/09 04:38, Fasiha Ashraf wrote:> Is this the issue that is affecting guest to guest throughput? Since I > have timer frequency 1000Hz and high resolution timer both enabled in > my dom0. Also i am using cup cores pinned for each VM an Dom0. Please > guide me some more about getting increased throughput between guests. >I haven''t specifically looked at the problem of domain<->domain performance. The quantization of time at 1kHz is a concern, but its not clear to me whether it comes from Linux, Xen or some interaction between them. J Connect more, do more and share more with Yahoo! India Mail. Learn more. http://in.overview.mail.yahoo.com/ _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jeremy Fitzhardinge
2009-Oct-06 05:30 UTC
Re: [Xen-devel] Re: [Xen-users] About profiling xen
On 10/05/09 22:01, Fasiha Ashraf wrote:> Which IDE is best suitable for inspecting xen code? Netbeans or > eclipse can be used? >I use emacs with cscope for navigating around. I don''t use IDEs much. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Is this guest to guest throuput issue resolved? What is affecting guest to guest throughput? Since I have timer frequency 1000Hz and high resolution timer both enabled in my dom0. Also i am using cpu cores pinned for each VM an Dom0. Please guide me some more about getting increased throughput between guests. Regards, Fasiha Ashraf --- On Wed, 30/9/09, Marco Tizzoni <marco.tizzoni@gmail.com> wrote: From: Marco Tizzoni <marco.tizzoni@gmail.com> Subject: Re: [Xen-devel] Re: [Xen-users] About profiling xen To: "Pasi Kärkkäinen" <pasik@iki.fi> Cc: "Fajar A." <fajar@fajar.net>, "Fasiha Ashraf" <feehapk@yahoo.co.in>, "xen" <xen-devel@lists.xensource.com>, xen-users@lists.xensource.com Date: Wednesday, 30 September, 2009, 11:40 AM On Wed, Sep 30, 2009 at 8:37 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:> On Wed, Sep 30, 2009 at 08:33:19AM +0200, Marco Tizzoni wrote: >> I experienced the same problem with netperf and with hping also. In >> general every tool that sends packet at fixed rates seems to suffer this >> problem. If you use the flood option ( sends packets as fast as possible - >> for example ''ping -f'' or ''hping3 -f'') instead of a fixed rate the rate >> become much higher. What I discovered is this issue should be related to >> Xen''s signal handling/delivery. >> Netperf use setitimer() to send packets at fixed rate. In my experience on >> dom0/U, with no load, Xen cannot go over 250 signals per second. Because >> netperf use signal wakeup the function that send packets I think your >> problem can be related to this issue. If you want to be sure, count the >> average number of packet per seconds. It should be about 250 packets/s >> (not more). > > > 250 per second.. sounds like kernel HZ value?Yes, infact I''m going to test it with 1000Khz. Stay tuned. :) m- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel Yahoo! India has a new look. Take a sneak peek http://in.yahoo.com/trynew _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users