Atsushi SAKAI
2008-Dec-11 08:02 UTC
[Xen-devel] [PATCH][RFC] remove PV/x86 warnings (time went backwards)
Hi, This patch intends to remove warnings for Time went backwards (negative delta) like follows. ========from Xen BZ 195 Timer ISR/0: Time went backwards: delta=-10048465 cpu_delta=9951535 shadow=706605145 0: 71640046095 ======= Currently (static u64) processed_system_time@time-xen.c uses for calculating jiffies and wall_clock. and the processed_system_time saves maximum vcpu time value of per_cpu(processed_system_time). At timer_interrupt@time-xen.c, they calculate "delta" compared to vcpu time(shadow). sometimes the value goes to negative. This is because vcpu time is varies.(scheduler etc.) This patch just removes the warnings of PV/x86, (Other way is loosen "delta" checking.(current checking is < -10msec) Or vcpu synchronization is needed.) This problem occurs on current linux-2.6.18-xen tree only. Not occured on pv_ops kernel. Of course, delta_cpu comparison keeps since it is just compares per_cpu variable only. ==for paravirt_ops =For paravirt_ops, time.c is completely changed. http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob;f=arch/x86/xen/time.c;h=c9f7cda48ed78ecbe1b421c540bbe2b777681400;hb=HEAD See also 2006/07(which uses delta) http://lists.xensource.com/archives/html/xen-devel/2006-07/msg00567.html 2007/03(which does not uses delta) http://lists.xensource.com/archives/html/xen-devel/2007-03/msg00554.html == Related Bugzilla =This patch related to following bugzilla. XenBugzilla: Many Timer ISR: Time went backwards logs http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=195 RedHat Bugzilla: Many "Timer ISR: Time went backwards" after booting xen kernel https://bugzilla.redhat.com/show_bug.cgi?id=222973 Signed-off-by: Atsushi SAKAI <sakaia@jp.fujitsu.com> Thanks Atsushi SAKAI _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tian, Kevin
2008-Dec-11 08:17 UTC
RE: [Xen-devel] [PATCH][RFC] remove PV/x86 warnings (time went backwards)
"Time went backward" warnings are indicator that something goes wrong. It''s not a solution to remove the warning while leaving actual reason unsolved. Which release are you trying? Does it happen on up-to-date xen unstable? Thanks, Kevin>From: Atsushi SAKAI >Sent: Thursday, December 11, 2008 4:03 PM > >Hi, > >This patch intends to remove warnings for Time went backwards >(negative delta) like follows. >========from Xen BZ 195 >Timer ISR/0: Time went backwards: delta=-10048465 cpu_delta=9951535 >shadow=706605145 0: 71640046095 >=======> >Currently (static u64) processed_system_time@time-xen.c > uses for calculating jiffies and wall_clock. >and the processed_system_time saves maximum vcpu time value of >per_cpu(processed_system_time). > >At timer_interrupt@time-xen.c, >they calculate "delta" compared to vcpu time(shadow). >sometimes the value goes to negative. >This is because vcpu time is varies.(scheduler etc.) > >This patch just removes the warnings of PV/x86, >(Other way is loosen "delta" checking.(current checking is < -10msec) > Or vcpu synchronization is needed.) > >This problem occurs on current linux-2.6.18-xen tree only. >Not occured on pv_ops kernel. > >Of course, delta_cpu comparison keeps >since it is just compares per_cpu variable only. > >==for paravirt_ops =>For paravirt_ops, time.c is completely changed. >http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=blob >;f=arch/x86/xen/time.c;h=c9f7cda48ed78ecbe1b421c540bbe2b7776814 >00;hb=HEAD > >See also >2006/07(which uses delta) >http://lists.xensource.com/archives/html/xen-devel/2006-07/msg0 >0567.html >2007/03(which does not uses delta) >http://lists.xensource.com/archives/html/xen-devel/2007-03/msg0 >0554.html > > >== Related Bugzilla =>This patch related to following bugzilla. > >XenBugzilla: >Many Timer ISR: Time went backwards logs >http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=195 > >RedHat Bugzilla: >Many "Timer ISR: Time went backwards" after booting xen kernel >https://bugzilla.redhat.com/show_bug.cgi?id=222973 > > >Signed-off-by: Atsushi SAKAI <sakaia@jp.fujitsu.com> > >Thanks >Atsushi SAKAI > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Dec-11 09:22 UTC
Re: [Xen-devel] [PATCH][RFC] remove PV/x86 warnings (time went backwards)
On 11/12/2008 08:02, "Atsushi SAKAI" <sakaia@jp.fujitsu.com> wrote:> > This patch intends to remove warnings for Time went backwards > (negative delta) like follows. > ========from Xen BZ 195 > Timer ISR/0: Time went backwards: delta=-10048465 cpu_delta=9951535 > shadow=706605145 0: 71640046095 > =======That might be considered skirting around the problem. Are users still seeing this with xen 3.3 and xen unstable? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2008-Dec-11 10:31 UTC
Re: [Xen-devel] [PATCH][RFC] remove PV/x86 warnings (time wentbackwards)
>>> Keir Fraser <keir.fraser@eu.citrix.com> 11.12.08 10:22 >>> >On 11/12/2008 08:02, "Atsushi SAKAI" <sakaia@jp.fujitsu.com> wrote: > >> >> This patch intends to remove warnings for Time went backwards >> (negative delta) like follows. >> ========from Xen BZ 195 >> Timer ISR/0: Time went backwards: delta=-10048465 cpu_delta=9951535 >> shadow=706605145 0: 71640046095 >> =======> >That might be considered skirting around the problem. Are users still seeing >this with xen 3.3 and xen unstable?Unfortunately yes, but rather seldom these days. We just got a fresh report about a larger IBM system suffering from it - while the initial report they sent had hundreds of milliseconds in them, the most up-to-date one we have shows that it''s just a few milliseconds above the -10ms boundary. So far I haven''t been able to figure what causes this. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Atsushi SAKAI
2008-Dec-11 11:00 UTC
Re: [Xen-devel] [PATCH][RFC] remove PV/x86 warnings (timewentbackwards)
Hi, Keir, Kevin, and Jan Thank you for various suggestions. One more questions on this. I think possible solution is Sync vcpu0 per_cpu(processed_system_time) to static u64 processed_system_time on PV kernel. It must be solved on hypervisor layer in this case? Thanks Atsushi SAKAI "Jan Beulich" <jbeulich@novell.com> wrote:> >>> Keir Fraser <keir.fraser@eu.citrix.com> 11.12.08 10:22 >>> > >On 11/12/2008 08:02, "Atsushi SAKAI" <sakaia@jp.fujitsu.com> wrote: > > > >> > >> This patch intends to remove warnings for Time went backwards > >> (negative delta) like follows. > >> ========from Xen BZ 195 > >> Timer ISR/0: Time went backwards: delta=-10048465 cpu_delta=9951535 > >> shadow=706605145 0: 71640046095 > >> =======> > > >That might be considered skirting around the problem. Are users still seeing > >this with xen 3.3 and xen unstable? > > Unfortunately yes, but rather seldom these days. We just got a fresh > report about a larger IBM system suffering from it - while the initial report > they sent had hundreds of milliseconds in them, the most up-to-date > one we have shows that it''s just a few milliseconds above the -10ms > boundary. So far I haven''t been able to figure what causes this. > > Jan > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Dec-11 11:03 UTC
Re: [Xen-devel] [PATCH][RFC] remove PV/x86 warnings (time wentbackwards)
On 11/12/2008 10:31, "Jan Beulich" <jbeulich@novell.com> wrote:>> That might be considered skirting around the problem. Are users still seeing >> this with xen 3.3 and xen unstable? > > Unfortunately yes, but rather seldom these days. We just got a fresh > report about a larger IBM system suffering from it - while the initial report > they sent had hundreds of milliseconds in them, the most up-to-date > one we have shows that it''s just a few milliseconds above the -10ms > boundary. So far I haven''t been able to figure what causes this.Perhaps we should expand the default acceptable jitter a bit? Still, I''d hoped that 10ms would be plenty! -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2008-Dec-11 11:12 UTC
Re: [Xen-devel] [PATCH][RFC] remove PV/x86 warnings (timewentbackwards)
>>> Keir Fraser <keir.fraser@eu.citrix.com> 11.12.08 12:03 >>> >On 11/12/2008 10:31, "Jan Beulich" <jbeulich@novell.com> wrote: > >>> That might be considered skirting around the problem. Are users still seeing >>> this with xen 3.3 and xen unstable? >> >> Unfortunately yes, but rather seldom these days. We just got a fresh >> report about a larger IBM system suffering from it - while the initial report >> they sent had hundreds of milliseconds in them, the most up-to-date >> one we have shows that it''s just a few milliseconds above the -10ms >> boundary. So far I haven''t been able to figure what causes this. > >Perhaps we should expand the default acceptable jitter a bit? Still, I''d >hoped that 10ms would be plenty!So would I (and really I find 10ms pretty wide already), which is why I never proposed an adjustment here, despite getting reports about these messages popping up every once in a while. And then again, it''s configurable on the command line and via sysctl, so if someone really just wants the messages go away, he can already do that. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel