I would appreciate if somebody could shed some light on the currently preferred setting of the clocksource for the pvops0 kernels, used both in Dom0 and DomU (I''m using pvops0 in DomU because it provides backends, e.g. network backend). I have a bad experience with running the Dom0 kernel (xen/stabale-2.6.31.6-based) with the default clocksource=xen, as it results in the system getting temporary "hangs", very short ones, but annoying. E.g. when one types fast on a keyboard, then every 10 seconds or so, the keystroke processing seems to be slowing down significantly for a second o so. Running the kernel with clocksource=jiffies eliminates the above problem but has a disadvantage of the clock drift in Dom0. This is not acceptable on my setup, where I don''t have any networking in Dom0, which means I cannot correct it via NTP. Interestingly the above problem didn''t seem to affect the Dom0 kernel based on xen/stable-2.6.32. Unfortunately I assume I''m no longer able to run this kernel when using Xen 3.4.2. Thanks, joanna. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 09/03/2010 10:47, "Joanna Rutkowska" <joanna@invisiblethingslab.com> wrote:> Interestingly the above problem didn''t seem to affect the Dom0 kernel > based on xen/stable-2.6.32. Unfortunately I assume I''m no longer able to > run this kernel when using Xen 3.4.2.You could run the 3.4.3 release candidates? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/09/2010 11:50 AM, Keir Fraser wrote:> On 09/03/2010 10:47, "Joanna Rutkowska" <joanna@invisiblethingslab.com> > wrote: > >> Interestingly the above problem didn''t seem to affect the Dom0 kernel >> based on xen/stable-2.6.32. Unfortunately I assume I''m no longer able to >> run this kernel when using Xen 3.4.2. > > You could run the 3.4.3 release candidates? >Let''s just say that after my recent experience with a 4.0.0 release candidate, I decided to use a stable version (or at least for the hypervisor) for now :) I really want to use on a production system. Perhaps you would say that in that case I should not be using a pvops0 kernel at all? But what would be a good alternative with all reasonable hardware support then? joanna. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/09/2010 11:54 AM, Joanna Rutkowska wrote:> On 03/09/2010 11:50 AM, Keir Fraser wrote: >> On 09/03/2010 10:47, "Joanna Rutkowska" <joanna@invisiblethingslab.com> >> wrote: >> >>> Interestingly the above problem didn''t seem to affect the Dom0 kernel >>> based on xen/stable-2.6.32. Unfortunately I assume I''m no longer able to >>> run this kernel when using Xen 3.4.2. >> >> You could run the 3.4.3 release candidates? >> > Let''s just say that after my recent experience with a 4.0.0 release > candidate, I decided to use a stable version (or at least for the > hypervisor) for now :) I really want to use on a production system. > > Perhaps you would say that in that case I should not be using a pvops0 > kernel at all? But what would be a good alternative with all reasonable > hardware support then? >...plus the xen/stable-2.6/.32 doesn''t have the pciback, which I happen to need in Dom0... j. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Mar 09, 2010 at 12:06:57PM +0100, Joanna Rutkowska wrote:> On 03/09/2010 11:54 AM, Joanna Rutkowska wrote: > > On 03/09/2010 11:50 AM, Keir Fraser wrote: > >> On 09/03/2010 10:47, "Joanna Rutkowska" <joanna@invisiblethingslab.com> > >> wrote: > >> > >>> Interestingly the above problem didn''t seem to affect the Dom0 kernel > >>> based on xen/stable-2.6.32. Unfortunately I assume I''m no longer able to > >>> run this kernel when using Xen 3.4.2. > >> > >> You could run the 3.4.3 release candidates? > >> > > Let''s just say that after my recent experience with a 4.0.0 release > > candidate, I decided to use a stable version (or at least for the > > hypervisor) for now :) I really want to use on a production system. > > > > Perhaps you would say that in that case I should not be using a pvops0 > > kernel at all? But what would be a good alternative with all reasonable > > hardware support then? > > > ...plus the xen/stable-2.6/.32 doesn''t have the pciback, which I happen > to need in Dom0... >Your options are listed here: http://wiki.xensource.com/xenwiki/XenDom0Kernels -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/09/2010 01:07 PM, Pasi Kärkkäinen wrote:> On Tue, Mar 09, 2010 at 12:06:57PM +0100, Joanna Rutkowska wrote: >> On 03/09/2010 11:54 AM, Joanna Rutkowska wrote: >>> On 03/09/2010 11:50 AM, Keir Fraser wrote: >>>> On 09/03/2010 10:47, "Joanna Rutkowska" <joanna@invisiblethingslab.com> >>>> wrote: >>>> >>>>> Interestingly the above problem didn''t seem to affect the Dom0 kernel >>>>> based on xen/stable-2.6.32. Unfortunately I assume I''m no longer able to >>>>> run this kernel when using Xen 3.4.2. >>>> >>>> You could run the 3.4.3 release candidates? >>>> >>> Let''s just say that after my recent experience with a 4.0.0 release >>> candidate, I decided to use a stable version (or at least for the >>> hypervisor) for now :) I really want to use on a production system. >>> >>> Perhaps you would say that in that case I should not be using a pvops0 >>> kernel at all? But what would be a good alternative with all reasonable >>> hardware support then? >>> >> ...plus the xen/stable-2.6/.32 doesn''t have the pciback, which I happen >> to need in Dom0... >> > > Your options are listed here: > http://wiki.xensource.com/xenwiki/XenDom0Kernels >Ok, but what about the original question about what is the preferred clocksource setting and perhaps some comments on various settings? Thanks, joanna. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Mar 09, 2010 at 12:06:57PM +0100, Joanna Rutkowska wrote:> On 03/09/2010 11:54 AM, Joanna Rutkowska wrote: > > On 03/09/2010 11:50 AM, Keir Fraser wrote: > >> On 09/03/2010 10:47, "Joanna Rutkowska" <joanna@invisiblethingslab.com> > >> wrote: > >> > >>> Interestingly the above problem didn''t seem to affect the Dom0 kernel > >>> based on xen/stable-2.6.32. Unfortunately I assume I''m no longer able to > >>> run this kernel when using Xen 3.4.2. > >> > >> You could run the 3.4.3 release candidates? > >> > > Let''s just say that after my recent experience with a 4.0.0 release > > candidate, I decided to use a stable version (or at least for the > > hypervisor) for now :) I really want to use on a production system. > > > > Perhaps you would say that in that case I should not be using a pvops0 > > kernel at all? But what would be a good alternative with all reasonable > > hardware support then? > > > ...plus the xen/stable-2.6/.32 doesn''t have the pciback, which I happen > to need in Dom0...You will be happy to know that I just finished that. git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git pv/pciback-2.6.32 or a copy of jeremy/xen/net with the pciback merged in: pv/merge.xen.next Or refer to the e-mail I have sent to Xen-Devel with details. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/09/2010 02:47 AM, Joanna Rutkowska wrote:> I would appreciate if somebody could shed some light on the currently > preferred setting of the clocksource for the pvops0 kernels, used both > in Dom0 and DomU (I''m using pvops0 in DomU because it provides backends, > e.g. network backend). >clocksource=xen is supposed to be the ideal clocksource, with no downsides.> I have a bad experience with running the Dom0 kernel > (xen/stabale-2.6.31.6-based) with the default clocksource=xen, as it > results in the system getting temporary "hangs", very short ones, but > annoying. E.g. when one types fast on a keyboard, then every 10 seconds > or so, the keystroke processing seems to be slowing down significantly > for a second o so. >There have been a number of reports about hiccups of varying degrees of severity when using clocksource=xen (and, interestingly, in KVM when using its very similar pv clock interface). I haven''t seen anyone mention keyboard interactive performance; the failure mode I''ve seen is disk IO hiccuping or even getting completely wedged. Unfortunately I haven''t made much headway in debugging it (or even reliably reproducing it). What is the hardware platform? How many CPUs are you using? Does pinning the vcpus to pcpus help?> Running the kernel with clocksource=jiffies eliminates the above problem > but has a disadvantage of the clock drift in Dom0. This is not > acceptable on my setup, where I don''t have any networking in Dom0, which > means I cannot correct it via NTP. > > Interestingly the above problem didn''t seem to affect the Dom0 kernel > based on xen/stable-2.6.32.That''s interesting; there''s no difference in the Xen-specific timekeeping parts of the kernel, but perhaps something else has changed in the way time is handled. I don''t remember if 3.4.2 ended up with tsc emulation, but if it does you might try enabling it.> Unfortunately I assume I''m no longer able to > run this kernel when using Xen 3.4.2. >Yes. 3.4.3 will support the newer kernels, but that isn''t released yet. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/10/2010 12:36 AM, Jeremy Fitzhardinge wrote:> On 03/09/2010 02:47 AM, Joanna Rutkowska wrote: >> I would appreciate if somebody could shed some light on the currently >> preferred setting of the clocksource for the pvops0 kernels, used both >> in Dom0 and DomU (I''m using pvops0 in DomU because it provides backends, >> e.g. network backend). >> > > clocksource=xen is supposed to be the ideal clocksource, with no downsides. > >> I have a bad experience with running the Dom0 kernel >> (xen/stabale-2.6.31.6-based) with the default clocksource=xen, as it >> results in the system getting temporary "hangs", very short ones, but >> annoying. E.g. when one types fast on a keyboard, then every 10 seconds >> or so, the keystroke processing seems to be slowing down significantly >> for a second o so. >> > > There have been a number of reports about hiccups of varying degrees of > severity when using clocksource=xen (and, interestingly, in KVM when > using its very similar pv clock interface). I haven''t seen anyone > mention keyboard interactive performance; the failure mode I''ve seen is > disk IO hiccuping or even getting completely wedged. Unfortunately I > haven''t made much headway in debugging it (or even reliably reproducing > it). > > What is the hardware platform?Intel Core 2 Duo, Intel PM45 chipset.> How many CPUs are you using?2 cores, all CPU-specific power management in BIOS disabled.> Does pinning the vcpus to pcpus help?I tried: xm vcpu-pin <dom> all all for all my running domains, and it didn''t change anything. But not sure if this is correct? Perhaps I should do: xm vcpu-pin <dom> 0 0 xm vcpu-pin <dom> 1 1 ? Perhaps there is a boot option to do the same system-wide?>> Running the kernel with clocksource=jiffies eliminates the above problem >> but has a disadvantage of the clock drift in Dom0. This is not >> acceptable on my setup, where I don''t have any networking in Dom0, which >> means I cannot correct it via NTP. >> >> Interestingly the above problem didn''t seem to affect the Dom0 kernel >> based on xen/stable-2.6.32. > > That''s interesting; there''s no difference in the Xen-specific > timekeeping parts of the kernel, but perhaps something else has changed > in the way time is handled. > >Interestingly, this keyboard hiccup effect does not occur in DomUs, only in Dom0. However, the DomUs clearly experience some scheduling problems -- e.g. sometime programs take a loooong time to start up, just like if their CPU time was miscalculated and they were constantly preempted, but xentop doesn''t show any high load anywhere. Occasionally this is visible in both Dom0''s and DomU''s dmesgs: Clocksource tsc unstable (delta = 62752834 ns) hrtimer: interrupt too slow, forcing clock min delta to 412700358 ns> I don''t remember if 3.4.2 ended up with tsc emulation, but if it does > you might try enabling it. >TSC emulation? But I''m running only PV guests. Not sure what do you mean by this, or how to enable it? j. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Clocksource tsc unstable (delta = 62752834 ns) > hrtimer: interrupt too slow, forcing clock min delta to 412700358 ns > > > I don''t remember if 3.4.2 ended up with tsc emulation, but if it does > > you might try enabling it. > > > > TSC emulation? But I''m running only PV guests. Not sure what do you > mean > by this, or how to enable it?I don''t think there is any TSC emulation (or more properly "rdtsc emulation") in 3.4.x, certainly not without a special Xen boot option. The TSC delta is troubling... if you hadn''t said you had turned off all power management, I would have guessed a problem with C-state management. Maybe Xen is discovering some power management capability not visible in BIOS settings? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/11/2010 01:07 AM, Dan Magenheimer wrote:>> Clocksource tsc unstable (delta = 62752834 ns) >> hrtimer: interrupt too slow, forcing clock min delta to 412700358 ns >> >>> I don''t remember if 3.4.2 ended up with tsc emulation, but if it does >>> you might try enabling it. >>> >> >> TSC emulation? But I''m running only PV guests. Not sure what do you >> mean >> by this, or how to enable it? > > I don''t think there is any TSC emulation (or more properly "rdtsc emulation") > in 3.4.x, certainly not without a special Xen boot option. > > The TSC delta is troubling...Actually I''m getting this also when booting Dom0 (pvops 2.6.31.6-based) with clocksource=jifffies: Clocksource tsc unstable (delta = 62528016 ns) But, there are not visible problems like kbd hiccup or process scheduling in VM (besides wallclock drift in Dom0 which is annoying).> if you hadn''t said you had turned > off all power management, I would have guessed a problem with > C-state management. Maybe Xen is discovering some power management > capability not visible in BIOS settings? >I''m not very much of an ACPI expert, so if there are any output I can send, just let me know. j. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> The TSC delta is troubling... if you hadn''t said you had turned > off all power management, I would have guessed a problem with > C-state management. Maybe Xen is discovering some power management > capability not visible in BIOS settings?Xen uses the architectural C state mechanism, bypassing ACPI. Use "max_cstate=0" Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> From: Ian Pratt [mailto:Ian.Pratt@eu.citrix.com] > Sent: Wednesday, March 10, 2010 5:22 PM > To: Dan Magenheimer; Joanna Rutkowska; Jeremy Fitzhardinge > Cc: Ian Pratt; xen-devel@lists.xensource.com > Subject: RE: [Xen-devel] A clocksource question > > > The TSC delta is troubling... if you hadn''t said you had turned > > off all power management, I would have guessed a problem with > > C-state management. Maybe Xen is discovering some power management > > capability not visible in BIOS settings? > > Xen uses the architectural C state mechanism, bypassing ACPI. > Use "max_cstate=0"Yes! The Core 2 Duo fooled me. There are some versions of Core 2 Duo ("Conroe") that don''t support C3-state and some ("Merom") that do support C3-state. And the code to "recover" from C3, which I think was added before 3.4 was released, has been observed to be very poor in its attempt to reset TSC after C3 to a reasonable value. I didn''t think it was THAT poor though! See: http://lists.xensource.com/archives/html/xen-devel/2009-10/msg01414.html (AFAIK, this is not fixed in 4.0, though the new default rdtsc emulation may mask the problem by ensuring TSC always moves forward, even across different processors.) So that may explain the TSC delta. Since the pv clocksource algorithm is dependent in part on the hardware TSC being synced reasonably well by Xen, that may also explain other clock strangeness. P.S. Joanna -- max_cstate=0 must be specified on the Xen boot line in grub.conf, not in the vm.cfg or grub.conf of the guest. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/11/2010 01:21 AM, Ian Pratt wrote:>> The TSC delta is troubling... if you hadn''t said you had turned >> off all power management, I would have guessed a problem with >> C-state management. Maybe Xen is discovering some power management >> capability not visible in BIOS settings? > > Xen uses the architectural C state mechanism, bypassing ACPI. > Use "max_cstate=0" >Tried it (as an argument to xen) and it didn''t seem to help. BTW, how does the clocksource=jiffies work on a pvops kernel in Dom0? j. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/10/2010 04:52 PM, Joanna Rutkowska wrote:> BTW, how does the clocksource=jiffies work on a pvops kernel in Dom0? >Not very well. clocksource=jiffies just sets up timer interrupts at approx 100ms intervals and assumes that''s 100ms. You get very low res timers and timekeeping. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 11/03/2010 02:06, Jeremy Fitzhardinge wrote:> On 03/10/2010 04:52 PM, Joanna Rutkowska wrote: >> BTW, how does the clocksource=jiffies work on a pvops kernel in Dom0? >> > > Not very well. clocksource=jiffies just sets up timer interrupts at > approx 100ms intervals and assumes that''s 100ms. You get very low res > timers and timekeeping. >So, how to explain that there is no wallclock drift *at all*, even in a long run -- uptime of a few days, in Dom0, when it uses the jiffies source? Anyway, I assume that the "xen" clock source is much more fine grained (1ms?) and so, maybe my kbd hiccups are caused by some code executed by the timer interrupt too frequently (maybe too much code executes per each timer interrupt, because of some other bug)? Just a though... j. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/10/2010 05:19 PM, Joanna Rutkowska wrote:> On 11/03/2010 02:06, Jeremy Fitzhardinge wrote: > >> On 03/10/2010 04:52 PM, Joanna Rutkowska wrote: >> >>> BTW, how does the clocksource=jiffies work on a pvops kernel in Dom0? >>> >>> >> Not very well. clocksource=jiffies just sets up timer interrupts at >> approx 100ms intervals and assumes that''s 100ms. You get very low res >> timers and timekeeping. >> >> > So, how to explain that there is no wallclock drift *at all*, even in a > long run -- uptime of a few days, in Dom0, when it uses the jiffies source? >You said earlier that you were seeing clock drift with clocksource=jiffies in dom0.> Anyway, I assume that the "xen" clock source is much more fine grained > (1ms?)The xen clocksource has nanosecond resolution. But clocksources are different from event sources, and so the ns resolution of time measurement doesn''t have much relationship to the timer precision (which is always going to use the xen event source, which is also ns resolution, but it will tend to fold together timer events which are closer than 50us).> and so, maybe my kbd hiccups are caused by some code executed by > the timer interrupt too frequently (maybe too much code executes per > each timer interrupt, because of some other bug)? Just a though... >Possibly, but doesn''t seem too likely. Can you tell if your hiccups are accompanied by bursts of timer interrupts in /proc/interrupts? There is another possibility, which is that the scheduler is getting confused by xen''s scheduler clock. Rather than just scheduling based on real time, we try to take into account time stolen from a vcpu so that it isn''t credited against the process (which may have had all its time stolen by another domain). But that could just be confusing things. Does this help? diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c index 0d3f07c..9029885 100644 --- a/arch/x86/xen/time.c +++ b/arch/x86/xen/time.c @@ -161,6 +161,9 @@ static void do_stolen_accounting(void) */ unsigned long long xen_sched_clock(void) { +#if 1 + return xen_clocksource_read(); +#else struct vcpu_runstate_info state; cycle_t now; u64 ret; @@ -190,6 +193,7 @@ unsigned long long xen_sched_clock(void) preempt_enable(); return ret; +#endif } J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 10/03/2010 22:13, "Joanna Rutkowska" <joanna@invisiblethingslab.com> wrote:>> I don''t remember if 3.4.2 ended up with tsc emulation, but if it does >> you might try enabling it. > > TSC emulation? But I''m running only PV guests. Not sure what do you mean > by this, or how to enable it?There is no TSC emulation for PV guests in 3.4.x. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/11/2010 05:55 AM, Jeremy Fitzhardinge wrote:> On 03/10/2010 05:19 PM, Joanna Rutkowska wrote: >> On 11/03/2010 02:06, Jeremy Fitzhardinge wrote: >> >>> On 03/10/2010 04:52 PM, Joanna Rutkowska wrote: >>> >>>> BTW, how does the clocksource=jiffies work on a pvops kernel in Dom0? >>>> >>>> >>> Not very well. clocksource=jiffies just sets up timer interrupts at >>> approx 100ms intervals and assumes that''s 100ms. You get very low res >>> timers and timekeeping. >>> >>> >> So, how to explain that there is no wallclock drift *at all*, even in a >> long run -- uptime of a few days, in Dom0, when it uses the jiffies >> source? >> > > You said earlier that you were seeing clock drift with > clocksource=jiffies in dom0. >Correct, sorry. With jiffies I do get a wallclock drift, yes.>> Anyway, I assume that the "xen" clock source is much more fine grained >> (1ms?) > > The xen clocksource has nanosecond resolution. But clocksources are > different from event sources, and so the ns resolution of time > measurement doesn''t have much relationship to the timer precision (which > is always going to use the xen event source, which is also ns > resolution, but it will tend to fold together timer events which are > closer than 50us). > >> and so, maybe my kbd hiccups are caused by some code executed by >> the timer interrupt too frequently (maybe too much code executes per >> each timer interrupt, because of some other bug)? Just a though... >> > > Possibly, but doesn''t seem too likely. Can you tell if your hiccups are > accompanied by bursts of timer interrupts in /proc/interrupts? > > There is another possibility, which is that the scheduler is getting > confused by xen''s scheduler clock. Rather than just scheduling based on > real time, we try to take into account time stolen from a vcpu so that > it isn''t credited against the process (which may have had all its time > stolen by another domain). But that could just be confusing things. > > Does this help? > > diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.cWill build a new kernel with this and let you know. j. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/11/2010 05:55 AM, Jeremy Fitzhardinge wrote:>> and so, maybe my kbd hiccups are caused by some code executed by >> the timer interrupt too frequently (maybe too much code executes per >> each timer interrupt, because of some other bug)? Just a though... >> > > Possibly, but doesn''t seem too likely. Can you tell if your hiccups are > accompanied by bursts of timer interrupts in /proc/interrupts? > > There is another possibility, which is that the scheduler is getting > confused by xen''s scheduler clock. Rather than just scheduling based on > real time, we try to take into account time stolen from a vcpu so that > it isn''t credited against the process (which may have had all its time > stolen by another domain). But that could just be confusing things. > > Does this help? > > diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c > index 0d3f07c..9029885 100644 > --- a/arch/x86/xen/time.c > +++ b/arch/x86/xen/time.c > @@ -161,6 +161,9 @@ static void do_stolen_accounting(void) > */ > unsigned long long xen_sched_clock(void) > { > +#if 1 > + return xen_clocksource_read(); > +#else > struct vcpu_runstate_info state; > cycle_t now; > u64 ret; > @@ -190,6 +193,7 @@ unsigned long long xen_sched_clock(void) > preempt_enable(); > > return ret; > +#endif > } >Nope, it didn''t. I think that the important clue is this message appearing in my dmesg (in Dom0 and also in DomUs): hrtimer: interrupt too slow, forcing clock min delta to 540150561 ns This is almost 0.5s (!) and I think this might explain my kbd hiccup. I wrote that I feel it every 10s or so, but when I was playing on my system without jiffies setting today, I saw this hiccup occurring much more often; in fact it was more of a "slow keyboard/system" feeling than a hiccup". Anyway, this is a comment from the function that displays this warning (kernel/hrtimer.c): /* * After 5 iteration''s attempts, we consider that hrtimer_interrupt() * is hanging, which could happen with something that slows the interrupt * such as the tracing. Then we force the clock reprogramming for each future * hrtimer interrupts to avoid infinite loops and use the min_delta_ns * threshold that we will overwrite. * The next tick event will be scheduled to 3 times we currently spend on * hrtimer_interrupt(). This gives a good compromise, the cpus will spend * 1/4 of their time to process the hrtimer interrupts. This is enough to * let it running without serious starvation. */ static inline void hrtimer_interrupt_hanging(struct clock_event_device *dev, ktime_t try_time) { force_clock_reprogram = 1; dev->min_delta_ns = (unsigned long)try_time.tv64 * 3; printk(KERN_WARNING "hrtimer: interrupt too slow, " "forcing clock min delta to %lu ns\n", dev->min_delta_ns); } joanna. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/11/2010 01:44 AM, Dan Magenheimer wrote:>> From: Ian Pratt [mailto:Ian.Pratt@eu.citrix.com] >> Sent: Wednesday, March 10, 2010 5:22 PM >> To: Dan Magenheimer; Joanna Rutkowska; Jeremy Fitzhardinge >> Cc: Ian Pratt; xen-devel@lists.xensource.com >> Subject: RE: [Xen-devel] A clocksource question >> >>> The TSC delta is troubling... if you hadn''t said you had turned >>> off all power management, I would have guessed a problem with >>> C-state management. Maybe Xen is discovering some power management >>> capability not visible in BIOS settings? >> >> Xen uses the architectural C state mechanism, bypassing ACPI. >> Use "max_cstate=0" > > Yes! The Core 2 Duo fooled me. There are some versions > of Core 2 Duo ("Conroe") that don''t support C3-state and > some ("Merom") that do support C3-state. And the code > to "recover" from C3, which I think was added before 3.4 > was released, has been observed to be very poor in its > attempt to reset TSC after C3 to a reasonable value. I > didn''t think it was THAT poor though! See: > http://lists.xensource.com/archives/html/xen-devel/2009-10/msg01414.html > > (AFAIK, this is not fixed in 4.0, though the new default > rdtsc emulation may mask the problem by ensuring TSC always > moves forward, even across different processors.) > > So that may explain the TSC delta. Since the pv clocksource > algorithm is dependent in part on the hardware TSC being > synced reasonably well by Xen, that may also explain > other clock strangeness. > > P.S. Joanna -- max_cstate=0 must be specified on the > Xen boot line in grub.conf, not in the vm.cfg or grub.conf > of the guest.Dan, I tried passing this argument to Xen, of course and it didn''t help. I still think it''s a Dom0-related problem rather than a hypervisor, because when I was using xen/stable-2.6.32-based kernel for a moment, I''m pretty sure it worked fine and I didn''t have any problem with the clocksource. On the same hypervisor, when I used xen/stable-2.6.31 kernel, the problem was present. (I had to switch back to 2.6.31 because, back then, the 2.6.32 didn''t have pciback, which was important for me; and recently I decided to downgrade to Xen 3.4.2, for its presumed stability, and so 2.6.31 is the only option now). joanna. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/12/2010 01:49 AM, Jeremy Fitzhardinge wrote:> On 03/11/2010 04:27 PM, Joanna Rutkowska wrote: >> I think that the important clue is this message appearing in my dmesg >> (in Dom0 and also in DomUs): >> >> hrtimer: interrupt too slow, forcing clock min delta to 540150561 ns >> >> This is almost 0.5s (!) and I think this might explain my kbd hiccup. I >> wrote that I feel it every 10s or so, but when I was playing on my >> system without jiffies setting today, I saw this hiccup occurring much >> more often; in fact it was more of a "slow keyboard/system" feeling than >> a hiccup". >> > > Yes, its definitely a good clue. It could point to a Xen scheduler > issue (I assume you don''t have any other busy domains when this is going > on), but that wouldn''t explain why it works OK with jiffies. Perhaps > there''s a problem with programming the timer so that sometimes it gets > missed and the system doesn''t get a kick until some subsequent event > (whereas with clocksource=jiffies there''ll always be a timer in the next > 100ms). >One detail: when I boot the system and there is only Dom0 running, everything seems ok. Only after I start at least one VM the problem becomes noticeable. But the other VM(s) is pretty much idle (as xentop shows). When I kill all the other VMs, so that Dom0 is alone again, the problem does *not* seem to go away, strangely... j. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/11/2010 04:27 PM, Joanna Rutkowska wrote:> I think that the important clue is this message appearing in my dmesg > (in Dom0 and also in DomUs): > > hrtimer: interrupt too slow, forcing clock min delta to 540150561 ns > > This is almost 0.5s (!) and I think this might explain my kbd hiccup. I > wrote that I feel it every 10s or so, but when I was playing on my > system without jiffies setting today, I saw this hiccup occurring much > more often; in fact it was more of a "slow keyboard/system" feeling than > a hiccup". >Yes, its definitely a good clue. It could point to a Xen scheduler issue (I assume you don''t have any other busy domains when this is going on), but that wouldn''t explain why it works OK with jiffies. Perhaps there''s a problem with programming the timer so that sometimes it gets missed and the system doesn''t get a kick until some subsequent event (whereas with clocksource=jiffies there''ll always be a timer in the next 100ms). J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi, i think i have the same Dom0 Problems as Joanna. My setup is: Xen 4.0.0-rc3-pre with some GFX-Passthrough Patches (btw: works great!) with "vtd=1 dom0_max_vcpus=6 iommu=pv" as cmdline for xen Dom0 Kernel is currently 2.6.33 using rebased Opensuse xen-patches (xen- patches-2.6.33-1) I also get - a slightly different - hrtimer message , but only in my Dom0 (as i have only Windows HVM as DomU) "hrtimer: interrupt took 2951458 ns" This message seems to appear when my DomU is booting. "clocksource=jiffies" as Dom0-Parameter doesn''t help. "max_cstate=0" as xen Parameter doesn''t seem to help either. The Patch from http://lists.xensource.com/archives/html/xen- devel/2010-03/msg00570.html doesn''t seem to help. My "Hiccups" occur - felt - "randomly", they take from 1 up to 5 seconds (not messured, more felt) and the only thing moving on the screen during that is the mouse pointer. If it weren''t for the moving pointer one could think the Maschine is completly frozen. The "Hiccups" are ONLY in the Dom0, its even so that i can perfectly work in the DomU, even when one of these long-term-hiccups occur. That all just in case another "affected" system my help you in your analsysis of this strange behaviour. Tell me if i can help testing. Greetings Tobias P.S.: Sorry for the non-threaded Follow-up, but i just subscribed to the list... _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/12/2010 01:38 AM, Joanna Rutkowska wrote:> On 03/12/2010 01:49 AM, Jeremy Fitzhardinge wrote: >> On 03/11/2010 04:27 PM, Joanna Rutkowska wrote: >>> I think that the important clue is this message appearing in my dmesg >>> (in Dom0 and also in DomUs): >>> >>> hrtimer: interrupt too slow, forcing clock min delta to 540150561 ns >>> >>> This is almost 0.5s (!) and I think this might explain my kbd hiccup. I >>> wrote that I feel it every 10s or so, but when I was playing on my >>> system without jiffies setting today, I saw this hiccup occurring much >>> more often; in fact it was more of a "slow keyboard/system" feeling than >>> a hiccup". >>> >> >> Yes, its definitely a good clue. It could point to a Xen scheduler >> issue (I assume you don''t have any other busy domains when this is going >> on), but that wouldn''t explain why it works OK with jiffies. Perhaps >> there''s a problem with programming the timer so that sometimes it gets >> missed and the system doesn''t get a kick until some subsequent event >> (whereas with clocksource=jiffies there''ll always be a timer in the next >> 100ms). >> > One detail: when I boot the system and there is only Dom0 running, > everything seems ok. Only after I start at least one VM the problem > becomes noticeable. But the other VM(s) is pretty much idle (as xentop > shows). > > When I kill all the other VMs, so that Dom0 is alone again, the problem > does *not* seem to go away, strangely... >Today I upgraded to xen 3.4.3-rc3 and tried the 2.6.32 kernel (the Micheal Young''s one) -- the same problem :( joanna. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/12/2010 11:02 PM, Jeremy Fitzhardinge wrote:> On 03/12/2010 01:56 PM, Tobias Geiger wrote: >> kbpc2:~# cat >> /sys/devices/system/clocksource/clocksource0/current_clocksource >> xen >> kbpc2:~# cat >> /sys/devices/system/clocksource/clocksource0/available_clocksource >> xen >> kbpc2:~# echo "jiffies" >> >>> /sys/devices/system/clocksource/clocksource0/current_clocksource >>> >> kbpc2:~# dmesg | tail -1 >> [ 7898.642404] Override clocksource jiffies is not HRT compatible. >> Cannot switch >> while in HRT/NOHZ mode >> kbpc2:~# >> >> Seems like it''s not possible to switch to jiffies - however: >> kbpc2:~# zcat /proc/config.gz | grep NO_HZ >> # CONFIG_NO_HZ is not set >> >> but CONFIG_SCHED_HRTICK=y >> > > Interesting. BTW, does NO_HZ change the behaviour of this? In > principle NO_HZ should be preferable, and particularly in a virtual > machine since it should limit the number of timer interrupts. > > Joanna, are you using NO_HZ? >Yes, I have: CONFIG_NO_HZ=y BTW, FWIW, I''m attaching my kernel config. It''s almost the same as the previously mentioned "Young''s Dec23" kernel (the only difference being some more recent pvops git patches applied and pcifront being enabled as a module). While trying to switch from xen source to jiffies source, I got the same message as Tobias: jiffies clocksource is not HRT compatible. Cannot switch while in HRT/NOHZ mode I can definitely say that this previously mentioned warning from hrtimer about interrupt being too slow is the marking of the start of all the troubles. E.g. when I run my Dom0 for some time without starting any other VMs, the system behaves good, and no warning in dmesg. Only after I start a VM, and probably depending on how heavy load it produces, the warning appears in Dom0 dmesg and the problem becomes "feelable". And, depending on what new "delta" is being chosen, the system is either still usable -- e.g. with a delta like in this case: hrtimer: interrupt too slow, forcing clock min delta to 82595226 ns which is around 80ms (still the very subtle hiccups can be observed when you press and hold a key for some time), or totally unusable, if one is unlucky, and the "safe" delta got chosen to be something around 0.5s (which interestingly happens most often in my case). The "clocksource tsc unstable" messages trouble me too BTW. They seem to appear before the hrtimer "interrupt too slow" messages. joanna. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/12/2010 10:01 PM, Joanna Rutkowska wrote:> On 03/12/2010 11:02 PM, Jeremy Fitzhardinge wrote: >> On 03/12/2010 01:56 PM, Tobias Geiger wrote: >>> kbpc2:~# cat >>> /sys/devices/system/clocksource/clocksource0/current_clocksource >>> xen >>> kbpc2:~# cat >>> /sys/devices/system/clocksource/clocksource0/available_clocksource >>> xen >>> kbpc2:~# echo "jiffies" >>> >>>> /sys/devices/system/clocksource/clocksource0/current_clocksource >>>> >>> kbpc2:~# dmesg | tail -1 >>> [ 7898.642404] Override clocksource jiffies is not HRT compatible. >>> Cannot switch >>> while in HRT/NOHZ mode >>> kbpc2:~# >>> >>> Seems like it''s not possible to switch to jiffies - however: >>> kbpc2:~# zcat /proc/config.gz | grep NO_HZ >>> # CONFIG_NO_HZ is not set >>> >>> but CONFIG_SCHED_HRTICK=y >>> >> >> Interesting. BTW, does NO_HZ change the behaviour of this? In >> principle NO_HZ should be preferable, and particularly in a virtual >> machine since it should limit the number of timer interrupts. >> >> Joanna, are you using NO_HZ? >> > Yes, I have: > CONFIG_NO_HZ=y > > BTW, FWIW, I''m attaching my kernel config. It''s almost the same as the > previously mentioned "Young''s Dec23" kernel (the only difference being > some more recent pvops git patches applied and pcifront being enabled as > a module). > > While trying to switch from xen source to jiffies source, I got the same > message as Tobias: > > jiffies clocksource is not HRT compatible. Cannot switch while in > HRT/NOHZ mode > > I can definitely say that this previously mentioned warning from hrtimer > about interrupt being too slow is the marking of the start of all the > troubles. E.g. when I run my Dom0 for some time without starting any > other VMs, the system behaves good, and no warning in dmesg. Only after > I start a VM, and probably depending on how heavy load it produces, the > warning appears in Dom0 dmesg and the problem becomes "feelable". And, > depending on what new "delta" is being chosen, the system is either > still usable -- e.g. with a delta like in this case: > > hrtimer: interrupt too slow, forcing clock min delta to 82595226 ns > > which is around 80ms (still the very subtle hiccups can be observed when > you press and hold a key for some time), or totally unusable, if one is > unlucky, and the "safe" delta got chosen to be something around 0.5s > (which interestingly happens most often in my case). > > The "clocksource tsc unstable" messages trouble me too BTW. They seem to > appear before the hrtimer "interrupt too slow" messages. >But this "tsc unstable" message does not seem to be so critical, e.g. I just started my system (xen clocksource) and started a VM, and got the usual hrtimer "slow interrupt" warning, and then all the symptoms started, but there no "tsc unstables" warning in dmesg. However, I previously missed another message, that seem to occur at the very beginning of the system life (before I started any other VMs): Marking TSC unstable due to TSC halts in idle j. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/12/2010 11:02 AM, Joanna Rutkowska wrote:> Today I upgraded to xen 3.4.3-rc3 and tried the 2.6.32 kernel (the > Micheal Young''s one) -- the same problem :( >Good, in a way. I can''t think of a reason it would change from .31 -> .32. Experiment of the moment: does changing the clocksource in /sys/devices/system/clocksource/clocksource0/current_clocksource reset the problem after killing the other domains? Ie, switch it from xen->jiffies->xen for a while. Thanks, J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Am Freitag 12 März 2010 22:24:54 schrieb Jeremy Fitzhardinge:> On 03/12/2010 11:02 AM, Joanna Rutkowska wrote: > > Today I upgraded to xen 3.4.3-rc3 and tried the 2.6.32 kernel (the > > Micheal Young''s one) -- the same problem :( > > Good, in a way. I can''t think of a reason it would change from .31 -> .32. > > Experiment of the moment: does changing the clocksource in > /sys/devices/system/clocksource/clocksource0/current_clocksource reset > the problem after killing the other domains? Ie, switch it from > xen->jiffies->xen for a while. >kbpc2:~# cat /sys/devices/system/clocksource/clocksource0/current_clocksource xen kbpc2:~# cat /sys/devices/system/clocksource/clocksource0/available_clocksource xen kbpc2:~# echo "jiffies">/sys/devices/system/clocksource/clocksource0/current_clocksourcekbpc2:~# dmesg | tail -1 [ 7898.642404] Override clocksource jiffies is not HRT compatible. Cannot switch while in HRT/NOHZ mode kbpc2:~# Seems like it''s not possible to switch to jiffies - however: kbpc2:~# zcat /proc/config.gz | grep NO_HZ # CONFIG_NO_HZ is not set but CONFIG_SCHED_HRTICK=y Greetings Tobias> Thanks, > J > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/12/2010 01:56 PM, Tobias Geiger wrote:> kbpc2:~# cat > /sys/devices/system/clocksource/clocksource0/current_clocksource > xen > kbpc2:~# cat > /sys/devices/system/clocksource/clocksource0/available_clocksource > xen > kbpc2:~# echo "jiffies" > >> /sys/devices/system/clocksource/clocksource0/current_clocksource >> > kbpc2:~# dmesg | tail -1 > [ 7898.642404] Override clocksource jiffies is not HRT compatible. Cannot switch > while in HRT/NOHZ mode > kbpc2:~# > > Seems like it''s not possible to switch to jiffies - however: > kbpc2:~# zcat /proc/config.gz | grep NO_HZ > # CONFIG_NO_HZ is not set > > but CONFIG_SCHED_HRTICK=y >Interesting. BTW, does NO_HZ change the behaviour of this? In principle NO_HZ should be preferable, and particularly in a virtual machine since it should limit the number of timer interrupts. Joanna, are you using NO_HZ? Thanks, J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Am Freitag 12 März 2010 23:02:38 schrieb Jeremy Fitzhardinge:> BTW, does NO_HZ change the behaviour of this? >Oh yes! It does :) Seems like thats the key, at least for the very long and really annoying "hiccups" Whats remaining are very short hiccups (some tenths of a second or so) - seems to be related to heavy disk load or... well i can''t tell exactly from the short time NO_HZ is active here. Thanks so far - this is a HUGE improvment, at least here! Greetings and have a nice Weekend! Tobias _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/12/2010 01:13 PM, Joanna Rutkowska wrote:> >> The "clocksource tsc unstable" messages trouble me too BTW. They seem to >> appear before the hrtimer "interrupt too slow" messages. >> >> > But this "tsc unstable" message does not seem to be so critical,They''re nothing to worry about.> e.g. I > just started my system (xen clocksource) and started a VM, and got the > usual hrtimer "slow interrupt" warning, and then all the symptoms > started, but there no "tsc unstables" warning in dmesg. > > However, I previously missed another message, that seem to occur at the > very beginning of the system life (before I started any other VMs): > > Marking TSC unstable due to TSC halts in idle >That doesn''t matter because it doesn''t use the tsc clocksource. The Xen clocksource uses the tsc hardware register, but it isn''t subject to the same constraints that the tsc clocksource is. The tsc clocksource calibration is probably getting confused by being on a vcpu. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/13/2010 12:48 AM, Jeremy Fitzhardinge wrote:> On 03/12/2010 01:13 PM, Joanna Rutkowska wrote: >> >>> The "clocksource tsc unstable" messages trouble me too BTW. They seem to >>> appear before the hrtimer "interrupt too slow" messages. >>> >>> >> But this "tsc unstable" message does not seem to be so critical, > > They''re nothing to worry about. > >> e.g. I >> just started my system (xen clocksource) and started a VM, and got the >> usual hrtimer "slow interrupt" warning, and then all the symptoms >> started, but there no "tsc unstables" warning in dmesg. >> >> However, I previously missed another message, that seem to occur at the >> very beginning of the system life (before I started any other VMs): >> >> Marking TSC unstable due to TSC halts in idle >> > > That doesn''t matter because it doesn''t use the tsc clocksource. The Xen > clocksource uses the tsc hardware register, but it isn''t subject to the > same constraints that the tsc clocksource is. The tsc clocksource > calibration is probably getting confused by being on a vcpu. >In the meantime found a similar thread on KVM: http://www.mail-archive.com/kvm@vger.kernel.org/msg22925.html and also a patch suggested there: http://www.mail-archive.com/kvm@vger.kernel.org/msg23491.html Don''t have time to try the patch right now, but courious to hear your opinion in the meantime... Thanks, joanna. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/13/2010 01:58 AM, Joanna Rutkowska wrote:> In the meantime found a similar thread on KVM: > http://www.mail-archive.com/kvm@vger.kernel.org/msg22925.html > > and also a patch suggested there: > > http://www.mail-archive.com/kvm@vger.kernel.org/msg23491.html > > Don''t have time to try the patch right now, but courious to hear your > opinion in the meantime... >I was just looking at that logic yesterday and was approaching a similar conclusion, so I''m glad Marcelo sorted it out. It looks like a variant of that patch got into 2.6.33 (41d2e4949377) , so I''ll look at backporting it. Actually it''s probably -stable material. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> It looks like a variant > of that patch got into 2.6.33 (41d2e4949377) >according to http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.33.y.git;a=commit;h=41d2e4949377 there is a /proc/timer_list - here is mine with 2.6.33 and CONFIG_NOHZ active: Timer List Version: v0.5 HRTIMER_MAX_CLOCK_BASES: 2 now at 39700078385698 nsecs cpu: 0 clock 0: .base: ffff880008013f08 .index: 0 .resolution: 1 nsecs .get_time: ktime_get_real .offset: 1268594403242758210 nsecs active timers: clock 1: .base: ffff880008013f48 .index: 1 .resolution: 1 nsecs .get_time: ktime_get .offset: 0 nsecs active timers: #0: <ffff8800ec2f1948>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, xend/14751 # expires at 39700088244281-39700088294281 nsecs [in 9858583 to 9908583 nsecs] #1: <ffff88011de87a78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, kwin/3976 # expires at 39700088517904-39700088567904 nsecs [in 10132206 to 10182206 nsecs] #2: <ffff880114271a78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, gkrellm/4029 # expires at 39700090903507-39700090953507 nsecs [in 12517809 to 12567809 nsecs] #3: <ffff880008013fe0>, tick_sched_timer, S:01, tick_nohz_stop_sched_tick, swapper/0 # expires at 39700319000000-39700319000000 nsecs [in 240614302 to 240614302 nsecs] #4: <ffff8801172f5a78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, plasma- desktop/3980 # expires at 39700375492906-39700375841905 nsecs [in 297107208 to 297456207 nsecs] #5: <ffff88010e1cda78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, gkrellm/4030 # expires at 39700513816021-39700514369020 nsecs [in 435430323 to 435983322 nsecs] #6: <ffff88010e1cfa78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, gkrellm/4031 # expires at 39700928500172-39700929452171 nsecs [in 850114474 to 851066473 nsecs] #7: <ffff8801eeeb1948>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, xend/2498 # expires at 39701136632456-39701138632455 nsecs [in 1058246758 to 1060246757 nsecs] #8: <ffff8801ecff7a78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, avahi- daemon/2643 # expires at 39701630089942-39701658854941 nsecs [in 1551704244 to 1580469243 nsecs] #9: <ffff8801edd05948>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, munin- node/2936 # expires at 39701811354852-39701813354851 nsecs [in 1732969154 to 1734969153 nsecs] #10: <ffff8801ead57a78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, ntpd/1863 # expires at 39805530083909-39805630083909 nsecs [in 105451698211 to 105551698211 nsecs] #11: <ffff8800da159948>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, rsyslogd/16096 # expires at 39911498142700-39911598142700 nsecs [in 211419757002 to 211519757002 nsecs] .expires_next : 39700088294281 nsecs .hres_active : 1 .nr_events : 17061828 .nr_retries : 3170 .nr_hangs : 0 .max_hang_time : 0 nsecs .nohz_mode : 2 .idle_tick : 39700077000000 nsecs .tick_stopped : 1 .idle_jiffies : 4334367373 .idle_calls : 54289068 .idle_sleeps : 34911450 .idle_entrytime : 39700077164622 nsecs .idle_waketime : 39700077116565 nsecs .idle_exittime : 39700074096546 nsecs .idle_sleeptime : 31394496198384 nsecs .last_jiffies : 4334367374 .next_jiffies : 4334367616 .idle_expires : 39700319000000 nsecs jiffies: 4334367375 cpu: 1 clock 0: .base: ffff880008026f08 .index: 0 .resolution: 1 nsecs .get_time: ktime_get_real .offset: 1268594403242758210 nsecs active timers: clock 1: .base: ffff880008026f48 .index: 1 .resolution: 1 nsecs .get_time: ktime_get .offset: 0 nsecs active timers: #0: <ffff880008026fe0>, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0 # expires at 39700079083333-39700079083333 nsecs [in 697635 to 697635 nsecs] #1: <ffff8801ebdb6468>, it_real_fn, S:01, do_setitimer, uml_switch/2130 # expires at 39700827086561-39700827086561 nsecs [in 748700863 to 748700863 nsecs] #2: <ffff88000396f948>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, dirmngr/16013 # expires at 39700984969090-39700985969086 nsecs [in 906583392 to 907583388 nsecs] #3: <ffffffff816c1838>, sched_rt_period_timer, S:01, enqueue_task_rt, swapper/1 # expires at 39701000000000-39701000000000 nsecs [in 921614302 to 921614302 nsecs] #4: <ffff8801ee5df948>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, mysqld/2346 # expires at 39702678123264-39702683123263 nsecs [in 2599737566 to 2604737565 nsecs] #5: <ffff8801eeea5948>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, xend/2496 # expires at 100000015827667689-100000015927667689 nsecs [in 99960315749281991 to 99960315849281991 nsecs] .expires_next : 39700079083333 nsecs .hres_active : 1 .nr_events : 15140240 .nr_retries : 964 .nr_hangs : 1 .max_hang_time : 3968056 nsecs .nohz_mode : 2 .idle_tick : 39699985083333 nsecs .tick_stopped : 0 .idle_jiffies : 4334367281 .idle_calls : 20003807 .idle_sleeps : 5904872 .idle_entrytime : 39700078089128 nsecs .idle_waketime : 39700076359045 nsecs .idle_exittime : 39700076361140 nsecs .idle_sleeptime : 37710292484734 nsecs .last_jiffies : 4334367374 .next_jiffies : 4334367375 .idle_expires : 39701700000000 nsecs jiffies: 4334367375 cpu: 2 clock 0: .base: ffff880008039f08 .index: 0 .resolution: 1 nsecs .get_time: ktime_get_real .offset: 1268594403242758210 nsecs active timers: clock 1: .base: ffff880008039f48 .index: 1 .resolution: 1 nsecs .get_time: ktime_get .offset: 0 nsecs active timers: #0: <ffff880008039fe0>, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0 # expires at 39700079166666-39700079166666 nsecs [in 780968 to 780968 nsecs] #1: <ffff8801098aba78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, amarok/4040 # expires at 39700121985119-39700122084118 nsecs [in 43599421 to 43698420 nsecs] #2: <ffff880114253a78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, klipper/4011 # expires at 39700334245968-39700335244967 nsecs [in 255860270 to 256859269 nsecs] #3: <ffff8801143f9a78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, kmix/4018 # expires at 39700351305204-39700353303203 nsecs [in 272919506 to 274917505 nsecs] #4: <ffff8801ee6bfa78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, kontact/4039 # expires at 39700432386829-39700433385828 nsecs [in 354001131 to 355000130 nsecs] #5: <ffff8801ecd63948>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, mysqld/2344 # expires at 39701067743916-39701068743915 nsecs [in 989358218 to 990358217 nsecs] #6: <ffff88011dd5ba78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, kded4/3950 # expires at 39701351300666-39701356298665 nsecs [in 1272914968 to 1277912967 nsecs] #7: <ffff8801098e1a78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, akonadiserver/4049 # expires at 39701849232912-39701855575911 nsecs [in 1770847214 to 1777190213 nsecs] #8: <ffff88012091b948>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, gpg- agent/3856 # expires at 39702757252653-39702760249600 nsecs [in 2678866955 to 2681863902 nsecs] #9: <ffff880109b03a78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, akonadi_control/4047 # expires at 39707457215355-39707467205354 nsecs [in 7378829657 to 7388819656 nsecs] #10: <ffff8801edd43eb8>, hrtimer_wakeup, S:01, do_nanosleep, cron/2170 # expires at 39718231470880-39718231520880 nsecs [in 18153085182 to 18153135182 nsecs] #11: <ffff8800ec3f9d28>, hrtimer_wakeup, S:01, futex_wait_queue_me, rsyslogd/12234 # expires at 39733019433875-39733019483875 nsecs [in 32941048177 to 32941098177 nsecs] #12: <ffff8801ee14f948>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, mdadm/2194 # expires at 40019833645109-40019933645109 nsecs [in 319755259411 to 319855259411 nsecs] .expires_next : 39700079166666 nsecs .hres_active : 1 .nr_events : 6718421 .nr_retries : 1069 .nr_hangs : 0 .max_hang_time : 0 nsecs .nohz_mode : 2 .idle_tick : 39700068166666 nsecs .tick_stopped : 0 .idle_jiffies : 4334367364 .idle_calls : 8721853 .idle_sleeps : 3768379 .idle_entrytime : 39700067749764 nsecs .idle_waketime : 39700076873579 nsecs .idle_exittime : 39700076874813 nsecs .idle_sleeptime : 37892101915060 nsecs .last_jiffies : 4334367364 .next_jiffies : 4334367500 .idle_expires : 39700203000000 nsecs jiffies: 4334367375 cpu: 3 clock 0: .base: ffff88000804cf08 .index: 0 .resolution: 1 nsecs .get_time: ktime_get_real .offset: 1268594403242758210 nsecs active timers: clock 1: .base: ffff88000804cf48 .index: 1 .resolution: 1 nsecs .get_time: ktime_get .offset: 0 nsecs active timers: #0: <ffff88000804cfe0>, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0 # expires at 39700079249999-39700079249999 nsecs [in 864301 to 864301 nsecs] #1: <ffff8801efc9f948>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, init/1 # expires at 39702931328063-39702936328062 nsecs [in 2852942365 to 2857942364 nsecs] .expires_next : 39700079249999 nsecs .hres_active : 1 .nr_events : 7019712 .nr_retries : 703 .nr_hangs : 0 .max_hang_time : 0 nsecs .nohz_mode : 2 .idle_tick : 39697932249999 nsecs .tick_stopped : 0 .idle_jiffies : 4334365228 .idle_calls : 15211478 .idle_sleeps : 11625422 .idle_entrytime : 39699938261592 nsecs .idle_waketime : 39698025114890 nsecs .idle_exittime : 39698025116159 nsecs .idle_sleeptime : 37962384222924 nsecs .last_jiffies : 4334367235 .next_jiffies : 4334367236 .idle_expires : 39698029000000 nsecs jiffies: 4334367375 cpu: 4 clock 0: .base: ffff88000805ff08 .index: 0 .resolution: 1 nsecs .get_time: ktime_get_real .offset: 1268594403242758210 nsecs active timers: clock 1: .base: ffff88000805ff48 .index: 1 .resolution: 1 nsecs .get_time: ktime_get .offset: 0 nsecs active timers: #0: <ffff88000805ffe0>, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0 # expires at 39700079333332-39700079333332 nsecs [in 947634 to 947634 nsecs] #1: <ffff8801ecd77948>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, mysqld/2345 # expires at 39700515448847-39700516448846 nsecs [in 437063149 to 438063148 nsecs] #2: <ffff8801ee069a78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, hald- addon-stor/2731 # expires at 39700757370965-39700759288964 nsecs [in 678985267 to 680903266 nsecs] #3: <ffff8801ee5b9a78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, hald- addon-stor/2732 # expires at 39700757368857-39700759355856 nsecs [in 678983159 to 680970158 nsecs] #4: <ffff8801eedcfa78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, krunner/4005 # expires at 39700965439836-39700966438835 nsecs [in 887054138 to 888053137 nsecs] #5: <ffff8801ee0ffa78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, hald- addon-stor/2730 # expires at 39701757375385-39701759293384 nsecs [in 1678989687 to 1680907686 nsecs] #6: <ffff8801ecc95948>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, winbindd/2445 # expires at 39704287532303-39704317532302 nsecs [in 4209146605 to 4239146604 nsecs] #7: <ffff88011dc59a78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, klauncher/3948 # expires at 39704522370930-39704532360929 nsecs [in 4443985232 to 4453975231 nsecs] #8: <ffff8801eed35a78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, hald/2691 # expires at 39708757725862-39708787723861 nsecs [in 8679340164 to 8709338163 nsecs] #9: <ffff880120af7a78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, console-kit-dae/3640 # expires at 39711757158054-39711787158053 nsecs [in 11678772356 to 11708772355 nsecs] #10: <ffff8801ec733468>, it_real_fn, S:01, do_setitimer, qmgr/2821 # expires at 39950643408313-39950643408313 nsecs [in 250565022615 to 250565022615 nsecs] #11: <ffff8801efd64868>, it_real_fn, S:01, do_setitimer, master/2812 # expires at 40010643498250-40010643498250 nsecs [in 310565112552 to 310565112552 nsecs] #12: <ffff8801ebe03eb8>, hrtimer_wakeup, S:01, do_nanosleep, atd/2149 # expires at 43215598718315-43215598768315 nsecs [in 3515520332617 to 3515520382617 nsecs] #13: <ffff8801ebdc2068>, it_real_fn, S:01, do_setitimer, pickup/15002 # expires at 45677643431329-45677643431329 nsecs [in 5977565045631 to 5977565045631 nsecs] .expires_next : 39700079333332 nsecs .hres_active : 1 .nr_events : 5181117 .nr_retries : 1352 .nr_hangs : 2 .max_hang_time : 31154440 nsecs .nohz_mode : 2 .idle_tick : 39698048333332 nsecs .tick_stopped : 0 .idle_jiffies : 4334365344 .idle_calls : 6885906 .idle_sleeps : 3874150 .idle_entrytime : 39700078338583 nsecs .idle_waketime : 39698051036847 nsecs .idle_exittime : 39698051038239 nsecs .idle_sleeptime : 37271606112519 nsecs .last_jiffies : 4334367375 .next_jiffies : 4334367411 .idle_expires : 39698691000000 nsecs jiffies: 4334367375 cpu: 5 clock 0: .base: ffff880008072f08 .index: 0 .resolution: 1 nsecs .get_time: ktime_get_real .offset: 1268594403242758210 nsecs active timers: clock 1: .base: ffff880008072f48 .index: 1 .resolution: 1 nsecs .get_time: ktime_get .offset: 0 nsecs active timers: #0: <ffff8801ee259a78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, cupsd/2838 # expires at 39700188554200-39700199554199 nsecs [in 110168502 to 121168501 nsecs] #1: <ffff88011437da78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, kwalletd/4132 # expires at 39701350568747-39701355563746 nsecs [in 1272183049 to 1277178048 nsecs] #2: <ffff880008072fe0>, tick_sched_timer, S:01, tick_nohz_stop_sched_tick, swapper/0 # expires at 39701688000000-39701688000000 nsecs [in 1609614302 to 1609614302 nsecs] #3: <ffff8801ec4c5d28>, hrtimer_wakeup, S:01, futex_wait_queue_me, bacula- fd/3492 # expires at 39709742894693-39709742944693 nsecs [in 9664508995 to 9664558995 nsecs] #4: <ffff8801142fda78>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, korgac/4019 # expires at 39748359758573-39748419730572 nsecs [in 48281372875 to 48341344874 nsecs] #5: <ffff8801ee15b948>, hrtimer_wakeup, S:01, schedule_hrtimeout_range, rsyslogd/2096 # expires at 122711318146941-122711418146941 nsecs [in 83011239761243 to 83011339761243 nsecs] .expires_next : 39700199554199 nsecs .hres_active : 1 .nr_events : 3946402 .nr_retries : 883 .nr_hangs : 2 .max_hang_time : 2960096 nsecs .nohz_mode : 2 .idle_tick : 39699958416665 nsecs .tick_stopped : 1 .idle_jiffies : 4334367254 .idle_calls : 5770603 .idle_sleeps : 3654532 .idle_entrytime : 39699957423004 nsecs .idle_waketime : 39699921692874 nsecs .idle_exittime : 39699921694130 nsecs .idle_sleeptime : 37835103396410 nsecs .last_jiffies : 4334367254 .next_jiffies : 4334368985 .idle_expires : 39701688000000 nsecs jiffies: 4334367375 Tick Device: mode: 1 Per CPU device: 0 Clock Event Device: xen max_delta_ns: 4294967295 min_delta_ns: 100000 mult: 1 shift: 0 mode: 3 next_event: 39700088294281 nsecs set_next_event: vcpuop_set_next_event set_mode: vcpuop_set_mode event_handler: hrtimer_interrupt Tick Device: mode: 1 Per CPU device: 1 Clock Event Device: xen max_delta_ns: 4294967295 min_delta_ns: 100000 mult: 1 shift: 0 mode: 3 next_event: 39700079083333 nsecs set_next_event: vcpuop_set_next_event set_mode: vcpuop_set_mode event_handler: hrtimer_interrupt Tick Device: mode: 1 Per CPU device: 2 Clock Event Device: xen max_delta_ns: 4294967295 min_delta_ns: 100000 mult: 1 shift: 0 mode: 3 next_event: 39700079166666 nsecs set_next_event: vcpuop_set_next_event set_mode: vcpuop_set_mode event_handler: hrtimer_interrupt Tick Device: mode: 1 Per CPU device: 3 Clock Event Device: xen max_delta_ns: 4294967295 min_delta_ns: 100000 mult: 1 shift: 0 mode: 3 next_event: 39700079249999 nsecs set_next_event: vcpuop_set_next_event set_mode: vcpuop_set_mode event_handler: hrtimer_interrupt Tick Device: mode: 1 Per CPU device: 4 Clock Event Device: xen max_delta_ns: 4294967295 min_delta_ns: 100000 mult: 1 shift: 0 mode: 3 next_event: 39700079333332 nsecs set_next_event: vcpuop_set_next_event set_mode: vcpuop_set_mode event_handler: hrtimer_interrupt Tick Device: mode: 1 Per CPU device: 5 Clock Event Device: xen max_delta_ns: 4294967295 min_delta_ns: 100000 mult: 1 shift: 0 mode: 3 next_event: 39700199554199 nsecs set_next_event: vcpuop_set_next_event set_mode: vcpuop_set_mode event_handler: hrtimer_interrupt This is after some uptime with HVM guest active. As the ".nr_retries" indicate i still experience small hiccups, not long ones (but definitly longer than 100ms) but still quite annoying. CONFIG_NOHZ definitly helps, but not enough. Greetings Tobias _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/13/2010 08:30 PM, Jeremy Fitzhardinge wrote:> On 03/13/2010 01:58 AM, Joanna Rutkowska wrote: >> In the meantime found a similar thread on KVM: >> http://www.mail-archive.com/kvm@vger.kernel.org/msg22925.html >> >> and also a patch suggested there: >> >> http://www.mail-archive.com/kvm@vger.kernel.org/msg23491.html >> >> Don''t have time to try the patch right now, but courious to hear your >> opinion in the meantime... >> > > I was just looking at that logic yesterday and was approaching a similar > conclusion, so I''m glad Marcelo sorted it out. It looks like a variant > of that patch got into 2.6.33 (41d2e4949377) , so I''ll look at > backporting it. Actually it''s probably -stable material. >FWIW, I have finally built and now I''m testing the recent xen/stable-2.6.32.x with the hrtimer commit and it seems to be working fine. No hiccups. Thanks, joanna. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi, i also compiled 2.6.32.9 from xen/stable , but so far i can''t see much improvement. Still many small hiccups in Dom0 One time (i think after HVM DomU creation) i get: "hrtimer: interrupt took 4045137 ns" in Dom0 dmesg I also upgraded to newest xen-unstable: xen_changeset : Wed Mar 17 09:18:34 2010 +0000 21041:066c3eead6ec Could it be that the missing HPET is causing all this trouble? I just noticed that in "xm dmesg" : (XEN) CPUIDLE: disabled due to no HPET. Force enable with \047cpuidle\047. I don''t bother really about CPUIDLE, but i should perhaps about HPET ? Is there a reason xen doesn''t enable HPET? Motherboard is Intel DX58SO Thanks for your input Greetings Tobias Am Dienstag 16 März 2010 07:01:44 schrieb Joanna Rutkowska:> FWIW, I have finally built and now I''m testing the recent > xen/stable-2.6.32.x with the hrtimer commit and it seems to be working > fine. No hiccups. >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/17/2010 02:35 PM, Tobias Geiger wrote:> Hi, > > i also compiled 2.6.32.9 from xen/stable , but so far i can''t see much > improvement. > > Still many small hiccups in Dom0 > One time (i think after HVM DomU creation) i get: > "hrtimer: interrupt took 4045137 ns" in Dom0 dmesg > > I also upgraded to newest xen-unstable: > xen_changeset : Wed Mar 17 09:18:34 2010 +0000 21041:066c3eead6ec > >FWIW, I''m running under Xen 3.4.3-rc3. joanna. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/17/2010 06:35 AM, Tobias Geiger wrote:> i also compiled 2.6.32.9 from xen/stable , but so far i can''t see much > improvement. >I''m not certain you got the relevent changeset; the "xen/stable" branch now has it, but I think I only updated it yesterday. xen/stable is now 2.6.32.10.> Still many small hiccups in Dom0 > One time (i think after HVM DomU creation) i get: > "hrtimer: interrupt took 4045137 ns" in Dom0 dmesg >That''s only 4ms. Not great, but not too bad.> I also upgraded to newest xen-unstable: > xen_changeset : Wed Mar 17 09:18:34 2010 +0000 21041:066c3eead6ec > > > Could it be that the missing HPET is causing all this trouble? I just noticed > that in "xm dmesg" : > > (XEN) CPUIDLE: disabled due to no HPET. Force enable with \047cpuidle\047. > > I don''t bother really about CPUIDLE, but i should perhaps about HPET ? > Is there a reason xen doesn''t enable HPET? Motherboard is Intel DX58SO >I don''t think that will be relevent. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On many machines, HPET is disabled by default, but can be enabled in the pre-boot BIOS configuration screens.> -----Original Message----- > From: Tobias Geiger [mailto:tobias.geiger@vido.info] > Sent: Wednesday, March 17, 2010 7:36 AM > To: Joanna Rutkowska > Cc: Jeremy Fitzhardinge; xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] A clocksource question > > Hi, > > i also compiled 2.6.32.9 from xen/stable , but so far i can''t see much > improvement. > > Still many small hiccups in Dom0 > One time (i think after HVM DomU creation) i get: > "hrtimer: interrupt took 4045137 ns" in Dom0 dmesg > > I also upgraded to newest xen-unstable: > xen_changeset : Wed Mar 17 09:18:34 2010 +0000 > 21041:066c3eead6ec > > > Could it be that the missing HPET is causing all this trouble? I just > noticed > that in "xm dmesg" : > > (XEN) CPUIDLE: disabled due to no HPET. Force enable with > \047cpuidle\047. > > I don''t bother really about CPUIDLE, but i should perhaps about HPET ? > Is there a reason xen doesn''t enable HPET? Motherboard is Intel DX58SO > > Thanks for your input > > Greetings > Tobias > > > > > > Am Dienstag 16 März 2010 07:01:44 schrieb Joanna Rutkowska: > > FWIW, I have finally built and now I''m testing the recent > > xen/stable-2.6.32.x with the hrtimer commit and it seems to be > working > > fine. No hiccups. > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/17/2010 05:20 PM, Jeremy Fitzhardinge wrote:> On 03/17/2010 06:35 AM, Tobias Geiger wrote: >> i also compiled 2.6.32.9 from xen/stable , but so far i can''t see much >> improvement. >> > > I''m not certain you got the relevent changeset; the "xen/stable" branch > now has it, but I think I only updated it yesterday. xen/stable is now > 2.6.32.10. > >Correct, I used the xen/stable-2.6.32.x and indeed the xen/stable back a few days ago did not have this hrtimer commit. Jeremy, I think there might be some confusion about xen/stable vs. xen/stable-2.6.32.x as from your earlier email [1] it looked like those should be the same things, while in fact they are not. j. [1] http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00162.html _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 03/17/2010 10:05 AM, Joanna Rutkowska wrote:> On 03/17/2010 05:20 PM, Jeremy Fitzhardinge wrote: > >> On 03/17/2010 06:35 AM, Tobias Geiger wrote: >> >>> i also compiled 2.6.32.9 from xen/stable , but so far i can''t see much >>> improvement. >>> >>> >> I''m not certain you got the relevent changeset; the "xen/stable" branch >> now has it, but I think I only updated it yesterday. xen/stable is now >> 2.6.32.10. >> >> >> > Correct, I used the xen/stable-2.6.32.x and indeed the xen/stable back a > few days ago did not have this hrtimer commit. > > Jeremy, I think there might be some confusion about xen/stable vs. > xen/stable-2.6.32.x as from your earlier email [1] it looked like those > should be the same things, while in fact they are not. >Keeping them in sync is manual, and I forgot in that case. Well, I could justify it as being conservative in case the change introduced a regression. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel