Xen devel - Jan 2012 - (XEN) Platform timer appears to have unexpectedly wrapped 10 or more times.

Hi,

Sometimes, seemingly randomly, long-running Xen domains, using clocksource
xen, have their clock shift by ~3000 or ~36000 seconds, often with the dom0
complaining about clocksource tsc. The number changes if the hypervisor is
explicitly told to use clocksource=pit, but it still happens. It
doesn''t
seem to be particularly hardware-specific.

See also http://bugs.debian.org/599161

I see from the archives that others have reported this problem here in
February last year, but I couldn''t find a resolution. Can anyone help?

(Please Cc: responses, I''m not subscribed.)

-- 
     2. That which causes joy or happiness.

Hi

I have the same problem with 4 machines (hp dl385, AMD), the problem appears one
time per machine and per month ...
No chance to re-produce it faster. On some other machine with another AMD
processor, I never had this problem. Olivier Hanesse
reported the problem on some Intel CPU and not on other processor.

I''m using debian squeeze, with 

xen-hypervisor-4.0-amd64                4.0.1-4
linux-image-2.6.32-5-xen-amd64          2.6.32-38 (dom0)

that last changes that I have done is to add cpuidle=0 to hypervisor command
line. stable since 3-4 weeks (...)

I have e test system that runs 20 paravirt. domus, with wheezy, with same
problematic HW, with
xen-hypervisor-4.1-amd64                4.1.2-2
linux-image-3.1.0-1-amd64               3.1.6-1 (dom0)
and this system never had the TSC jump problem. but test system are test system
....

I would be happy if you could reproduce the problem faster a me, so testing new
xen / dom0 version could be
easier ...

thanks and regards

Philippe



> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of Josip Rodin
> Sent: Wednesday, January 04, 2012 6:24 PM
> To: xen-devel@lists.xensource.com
> Subject: [Xen-devel] (XEN) Platform timer appears to have unexpectedly
> wrapped 10 or more times.
> 
> Hi,
> 
> Sometimes, seemingly randomly, long-running Xen domains, using
> clocksource xen, have their clock shift by ~3000 or ~36000 seconds, often
> with the dom0 complaining about clocksource tsc. The number changes if the
> hypervisor is explicitly told to use clocksource=pit, but it still happens.
It
> doesn''t seem to be particularly hardware-specific.
> 
> See also http://bugs.debian.org/599161
> 
> I see from the archives that others have reported this problem here in
> February last year, but I couldn''t find a resolution. Can anyone
help?
> 
> (Please Cc: responses, I''m not subscribed.)
> 
> --
>      2. That which causes joy or happiness.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

I have the same problem. Do you use ntpd? Which version? I suspect some 
link between the problem and ntpd.

Andreas

On 05.01.2012 10:39, Philippe.Simonet@swisscom.com
wrote:
> Hi
>
> I have the same problem with 4 machines (hp dl385, AMD), the problem
appears one time per machine and per month ...
> No chance to re-produce it faster. On some other machine with another AMD
processor, I never had this problem. Olivier Hanesse
> reported the problem on some Intel CPU and not on other processor.
>
> I''m using debian squeeze, with
>
> xen-hypervisor-4.0-amd64                4.0.1-4
> linux-image-2.6.32-5-xen-amd64          2.6.32-38 (dom0)
>
> that last changes that I have done is to add cpuidle=0 to hypervisor
command line. stable since 3-4 weeks (...)
>
> I have e test system that runs 20 paravirt. domus, with wheezy, with same
problematic HW, with
> xen-hypervisor-4.1-amd64                4.1.2-2
> linux-image-3.1.0-1-amd64               3.1.6-1 (dom0)
> and this system never had the TSC jump problem. but test system are test
system ....
>
> I would be happy if you could reproduce the problem faster a me, so testing
new xen / dom0 version could be
> easier ...
>
> thanks and regards
>
> Philippe
>
>
>
>
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>> bounces@lists.xensource.com] On Behalf Of Josip Rodin
>> Sent: Wednesday, January 04, 2012 6:24 PM
>> To: xen-devel@lists.xensource.com
>> Subject: [Xen-devel] (XEN) Platform timer appears to have unexpectedly
>> wrapped 10 or more times.
>>
>> Hi,
>>
>> Sometimes, seemingly randomly, long-running Xen domains, using
>> clocksource xen, have their clock shift by ~3000 or ~36000 seconds,
often
>> with the dom0 complaining about clocksource tsc. The number changes if
the
>> hypervisor is explicitly told to use clocksource=pit, but it still
happens. It
>> doesn''t seem to be particularly hardware-specific.
>>
>> See also http://bugs.debian.org/599161
>>
>> I see from the archives that others have reported this problem here in
>> February last year, but I couldn''t find a resolution. Can
anyone help?
>>
>> (Please Cc: responses, I''m not subscribed.)
>>
>> --
>>       2. That which causes joy or happiness.
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

On Tue, Apr 10, 2012 at 02:33:18PM +0200, Andreas Kinzler
wrote:
> I have the same problem. Do you use ntpd? Which version? I suspect
> some link between the problem and ntpd.
But ntpd is still a userland program, shouldn''t the kernel exert the
ultimate control over who gets to screw with the system''s clock source?

In any case, I did a quick grep of ntpd sources and found references to
RDTSC only in the Windows port, so the link doesn''t seem obvious.

-- 
     2. That which causes joy or happiness.

> On Tue, Apr 10, 2012 at 02:33:18PM +0200, Andreas Kinzler wrote:
>> I have the same problem. Do you use ntpd? Which version? I suspect
>> some link between the problem and ntpd.
> But ntpd is still a userland program, shouldn''t the kernel exert
the
> ultimate control over who gets to screw with the system''s clock
source?
> In any case, I did a quick grep of ntpd sources and found references to
> RDTSC only in the Windows port, so the link doesn''t seem obvious.
Right, there is no obvious link (but which hard to reproduce bug is 
obvious?) but see 
"http://lists.xen.org/archives/html/xen-devel/2011-10/msg01269.html". 
There seems to be some connection.

Regards Andreas

On Tue, Apr 10, 2012 at 03:59:42PM +0200, Andreas Kinzler
wrote:
> >On Tue, Apr 10, 2012 at 02:33:18PM +0200, Andreas Kinzler wrote:
> >>I have the same problem. Do you use ntpd? Which version? I suspect
> >>some link between the problem and ntpd.
> >But ntpd is still a userland program, shouldn''t the kernel
exert the
> >ultimate control over who gets to screw with the system''s
clock source?
> >In any case, I did a quick grep of ntpd sources and found references to
> >RDTSC only in the Windows port, so the link doesn''t seem
obvious.
> 
> Right, there is no obvious link (but which hard to reproduce bug is
> obvious?) but see
>
"http://lists.xen.org/archives/html/xen-devel/2011-10/msg01269.html".
> There seems to be some connection.
From that (garbled) log, I don''t see it - you didn''t even post
the matching
kernel log message with timestamp, so it''s impossible to see if and
when
it coincided when the ntpd messages.

Over here we''re running ntpd, obviously, in dom0 and in domU, but the
only
observed interaction of this problem with ntpd is that they become useless
with this kind of a major drift.

-- 
     2. That which causes joy or happiness.