Dear fellow Xen''ers! I''m having a problem with my new Xen-enabled colocation server.. I hope ( at least ;) ) one of you guys know how to fix this problem.. For a certain amount of time ( 1-2 days ) the domU runs fine.. until the console starts spitting out messages like these: [80182.004456] clocksource/0: Time went backwards: delta=-6917292717540055641 shadow=80182004054805 offset=400471 [80182.004507] clocksource/0: Time went backwards: delta=-6917292717540005396 shadow=80182004054805 offset=450712 [80182.004762] clocksource/0: Time went backwards: delta=-6917292717539750360 shadow=80182004054805 offset=705812 [80182.004842] clocksource/0: Time went backwards: delta=-6917292717539670405 shadow=80182004054805 offset=785744 [80182.004918] clocksource/0: Time went backwards: delta=-6917292717539594298 shadow=80182004054805 offset=861851 The only action left is an xm shutdown, after which an xm create boots up the machine fine for the next couple of days.. I''m running Ubuntu 7.10, with the stock xen kernel ( 2.6.22-13-xen ). An ubuntu bug has already been reported and appended to by multiple persons here: https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.22/+bug/146924 Anyone have any ideas/patches/hints/.../anything that can help? Thanks a lot guys! Kind regards, Bart Verwilst _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Just happened again... Nobody has an idea what might be happening? This is obviously some kind of bug.. :) Thanks! Bart ----- Original Message ----- From: "Lists" <lists@verwilst.be> To: Xen-users@lists.xensource.com Sent: Wednesday, October 10, 2007 12:08:09 AM (GMT+0100) Europe/Berlin Subject: [Xen-users] clocksource/0: Time went backwards Dear fellow Xen''ers! I''m having a problem with my new Xen-enabled colocation server.. I hope ( at least ;) ) one of you guys know how to fix this problem.. For a certain amount of time ( 1-2 days ) the domU runs fine.. until the console starts spitting out messages like these: [80182.004456] clocksource/0: Time went backwards: delta=-6917292717540055641 shadow=80182004054805 offset=400471 [80182.004507] clocksource/0: Time went backwards: delta=-6917292717540005396 shadow=80182004054805 offset=450712 [80182.004762] clocksource/0: Time went backwards: delta=-6917292717539750360 shadow=80182004054805 offset=705812 [80182.004842] clocksource/0: Time went backwards: delta=-6917292717539670405 shadow=80182004054805 offset=785744 [80182.004918] clocksource/0: Time went backwards: delta=-6917292717539594298 shadow=80182004054805 offset=861851 The only action left is an xm shutdown, after which an xm create boots up the machine fine for the next couple of days.. I''m running Ubuntu 7.10, with the stock xen kernel ( 2.6.22-13-xen ). An ubuntu bug has already been reported and appended to by multiple persons here: https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.22/+bug/146924 Anyone have any ideas/patches/hints/.../anything that can help? Thanks a lot guys! Kind regards, Bart Verwilst _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Bart,
I haven''t had this happen but I have seen a lot of references to
similar problems on the mailing list archive. I would start there and
see if you can find your fix there.
Shane
Bart Verwilst wrote:> Just happened again...
> Nobody has an idea what might be happening? This is obviously some kind of
bug.. :)
>
> Thanks!
>
> Bart
>
> ----- Original Message -----
> From: "Lists" <lists@verwilst.be>
> To: Xen-users@lists.xensource.com
> Sent: Wednesday, October 10, 2007 12:08:09 AM (GMT+0100) Europe/Berlin
> Subject: [Xen-users] clocksource/0: Time went backwards
>
> Dear fellow Xen''ers!
>
> I''m having a problem with my new Xen-enabled colocation server.. I
hope ( at least ;) ) one of you guys know how to fix this problem..
>
> For a certain amount of time ( 1-2 days ) the domU runs fine.. until the
console starts spitting out messages like these:
>
> [80182.004456] clocksource/0: Time went backwards:
delta=-6917292717540055641 shadow=80182004054805 offset=400471
> [80182.004507] clocksource/0: Time went backwards:
delta=-6917292717540005396 shadow=80182004054805 offset=450712
> [80182.004762] clocksource/0: Time went backwards:
delta=-6917292717539750360 shadow=80182004054805 offset=705812
> [80182.004842] clocksource/0: Time went backwards:
delta=-6917292717539670405 shadow=80182004054805 offset=785744
> [80182.004918] clocksource/0: Time went backwards:
delta=-6917292717539594298 shadow=80182004054805 offset=861851
>
> The only action left is an xm shutdown, after which an xm create boots up
the machine fine for the next couple of days..
>
> I''m running Ubuntu 7.10, with the stock xen kernel ( 2.6.22-13-xen
).
>
> An ubuntu bug has already been reported and appended to by multiple persons
here:
>
> https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.22/+bug/146924
>
> Anyone have any ideas/patches/hints/.../anything that can help?
>
> Thanks a lot guys!
>
> Kind regards,
>
> Bart Verwilst
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Well, i''ve found this link earlier:
http://lists.xensource.com/archives/html/xen-devel/2006-03/msg00442.html , but
it seems the code has changed quite a bit since then..
Big parts of the patch seem integrated already though..
For example:
Current patch, in use in my running kernel:
+ if ((blocked > 0) && (delta_cpu > 0)) {
+ delta_cpu -= blocked;
+ if (unlikely(delta_cpu < 0))
+ blocked += delta_cpu; /* clamp local-time progress */
+ do_div(blocked, NS_PER_TICK);
+ per_cpu(processed_blocked_time, cpu) += blocked * NS_PER_TICK;
+ per_cpu(processed_system_time, cpu) += blocked * NS_PER_TICK;
Patch on the link provided above:
if (stolen > 0) {
delta_cpu -= stolen;
+ if (unlikely(delta_cpu < 0)) {
+ stolen += delta_cpu;
+ delta_cpu = blocked = 0;
+ }
do_div(stolen, NS_PER_TICK);
per_cpu(processed_stolen_time, cpu) += stolen * NS_PER_TICK;
per_cpu(processed_system_time, cpu) += stolen * NS_PER_TICK;
As you can see, quite different.. "delta_cpu = blocked = 0;" is
missing, but might be handled with the rest of the changes in the new code..
Another possibility is that this only applies for i386 (i''m running on
x86_64.. )..
I''ve looked into the patch, and it seems to only patch file
arch/i386/kernel/time-xen.c .
There isn''t a x86_64 equivalent.. ( arch/x86_64/kernel/time-xen.c
doesn''t exist ) Is this normal? Could this be the cause?
Thanks a lot for your reply!
Kind regards,
Bart Verwilst
----- Original Message -----
From: "Shane D. Johnson" <SDJ@RasmussenEquipment.com>
To: "Bart Verwilst" <lists@verwilst.be>
Cc: Xen-users@lists.xensource.com
Sent: Thursday, October 11, 2007 1:05:39 AM (GMT+0100) Europe/Berlin
Subject: Re: [Xen-users] clocksource/0: Time went backwards
Bart,
I haven''t had this happen but I have seen a lot of references to
similar problems on the mailing list archive. I would start there and
see if you can find your fix there.
Shane
Bart Verwilst wrote:> Just happened again...
> Nobody has an idea what might be happening? This is obviously some kind of
bug.. :)
>
> Thanks!
>
> Bart
>
> ----- Original Message -----
> From: "Lists" <lists@verwilst.be>
> To: Xen-users@lists.xensource.com
> Sent: Wednesday, October 10, 2007 12:08:09 AM (GMT+0100) Europe/Berlin
> Subject: [Xen-users] clocksource/0: Time went backwards
>
> Dear fellow Xen''ers!
>
> I''m having a problem with my new Xen-enabled colocation server.. I
hope ( at least ;) ) one of you guys know how to fix this problem..
>
> For a certain amount of time ( 1-2 days ) the domU runs fine.. until the
console starts spitting out messages like these:
>
> [80182.004456] clocksource/0: Time went backwards:
delta=-6917292717540055641 shadow=80182004054805 offset=400471
> [80182.004507] clocksource/0: Time went backwards:
delta=-6917292717540005396 shadow=80182004054805 offset=450712
> [80182.004762] clocksource/0: Time went backwards:
delta=-6917292717539750360 shadow=80182004054805 offset=705812
> [80182.004842] clocksource/0: Time went backwards:
delta=-6917292717539670405 shadow=80182004054805 offset=785744
> [80182.004918] clocksource/0: Time went backwards:
delta=-6917292717539594298 shadow=80182004054805 offset=861851
>
> The only action left is an xm shutdown, after which an xm create boots up
the machine fine for the next couple of days..
>
> I''m running Ubuntu 7.10, with the stock xen kernel ( 2.6.22-13-xen
).
>
> An ubuntu bug has already been reported and appended to by multiple persons
here:
>
> https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.22/+bug/146924
>
> Anyone have any ideas/patches/hints/.../anything that can help?
>
> Thanks a lot guys!
>
> Kind regards,
>
> Bart Verwilst
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Does ntpd allow the time to go backwards? However, i''ve found that the domU where i have been having most troubles with, doesn''t run ntpd. ( I set independent_wallclock in every domU anyways ), I''ve just enabled ntpd, and going to see what happens! Thanks Samuel! Kind regards, Bart ----- Original Message ----- From: "Samuel Thibault" <samuel.thibault@ens-lyon.org> To: "Shane D. Johnson" <SDJ@RasmussenEquipment.com> Cc: "Bart Verwilst" <lists@verwilst.be>, Xen-users@lists.xensource.com Sent: Thursday, October 11, 2007 1:36:44 AM (GMT+0100) Europe/Berlin Subject: Re: [Xen-users] clocksource/0: Time went backwards Shane D. Johnson, le Wed 10 Oct 2007 17:05:39 -0600, a écrit :> I haven''t had this happen but I have seen a lot of references to > similar problems on the mailing list archive. I would start there and > see if you can find your fix there.I also had this issue with Mach as domU. It looked to me like this is due to the NTP daemon running in dom0 that makes the time backward so as to align to NTP servers. Samuel _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Anyways, it just happened again, total freeze after those messages.. Looking for the problem on Google doesn''t really pay off either since i start encountering my own messages more and more :p So i guess ntpd(ate) isn''t responsible for this... What else can it be? There must be _someone_ who encountered and worked-around/fixed this? Pretty please with whipped cream and cherries on top Kind regards, Bart Verwilst ----- Original Message ----- From: "Samuel Thibault" <samuel.thibault@ens-lyon.org> To: "Bart Verwilst" <lists@verwilst.be> Cc: Xen-users@lists.xensource.com Sent: Thursday, October 11, 2007 11:01:44 AM (GMT+0100) Europe/Berlin Subject: Re: [Xen-users] clocksource/0: Time went backwards Bart Verwilst, le Thu 11 Oct 2007 10:07:55 +0200, a écrit :> Does ntpd allow the time to go backwards?In case the time is shifted a lot, yes. Samuel _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Stall the time? Could you elaborate on that please? Thanks! ----- Original Message ----- From: "Samuel Thibault" <samuel.thibault@ens-lyon.org> To: "Bart Verwilst" <lists@verwilst.be> Cc: Xen-users@lists.xensource.com Sent: Thursday, October 11, 2007 2:55:43 PM (GMT+0100) Europe/Berlin Subject: Re: [Xen-users] clocksource/0: Time went backwards Bart Verwilst, le Thu 11 Oct 2007 14:50:48 +0200, a écrit :> So i guess ntpd(ate) isn''t responsible for this... What else can it be? > There must be _someone_ who encountered and worked-around/fixed this?Well, the work around I made in Mach is just to stall the time until the hypervisor-provided time gets back in sync. Samuel _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Oh.. i was hoping for some simple sysctl tweak or easy patch :) Bummer :( ----- Original Message ----- From: "Samuel Thibault" <samuel.thibault@ens-lyon.org> To: "Bart Verwilst" <lists@verwilst.be> Cc: Xen-users@lists.xensource.com Sent: Thursday, October 11, 2007 3:02:49 PM (GMT+0100) Europe/Berlin Subject: Re: [Xen-users] clocksource/0: Time went backwards Bart Verwilst, le Thu 11 Oct 2007 14:59:31 +0200, a écrit :> Stall the time? > Could you elaborate on that please?In the Mach kernel''s time function, I record the timestamp of the last call. If the new timestamp from Xen is older than that one (i.e. a time shift occured), I keep returning the time of the recorded timestemp, until the timestamp from Xen becomes newer than the recorded one. Samuel _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Well, i just noticed this patch: http://lists.xensource.com/archives/html/xen-changelog/2007-10/msg00013.html Well, /proc/acpi/processor/CPU0/throttling says "<not supported>", and there seem to be no other frequency-related items, so i guess my server isn''t scaling his cpu''s... ----- Original Message ----- From: "Samuel Thibault" <samuel.thibault@ens-lyon.org> To: "Bart Verwilst" <lists@verwilst.be> Cc: Xen-users@lists.xensource.com Sent: Thursday, October 11, 2007 3:19:37 PM (GMT+0100) Europe/Berlin Subject: Re: [Xen-users] clocksource/0: Time went backwards Bart Verwilst, le Thu 11 Oct 2007 15:12:19 +0200, a écrit :> Oh.. i was hoping for some simple sysctl tweak or easy patch :) > Bummer :(Oh, actually, the problem you''re getting might be related to CPU frequency changes, try to disable that as well. Samuel _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Another thing i just tried is this: # cat /sys/devices/system/clocksource/clocksource0/current_clocksource xen # cat /sys/devices/system/clocksource/clocksource0/available_clocksource xen jiffies # echo jiffies > /sys/devices/system/clocksource/clocksource0/current_clocksource In dmesg: [57990.836097] Time: jiffies clocksource has been installed. Do you think this might work? Will it break something else? Is this a good idea? Thanks! Bart Verwilst ----- Original Message ----- From: "Samuel Thibault" <samuel.thibault@ens-lyon.org> To: "Bart Verwilst" <lists@verwilst.be> Cc: Xen-users@lists.xensource.com Sent: Thursday, October 11, 2007 3:19:37 PM (GMT+0100) Europe/Berlin Subject: Re: [Xen-users] clocksource/0: Time went backwards Bart Verwilst, le Thu 11 Oct 2007 15:12:19 +0200, a écrit :> Oh.. i was hoping for some simple sysctl tweak or easy patch :) > Bummer :(Oh, actually, the problem you''re getting might be related to CPU frequency changes, try to disable that as well. Samuel _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
In the meantime i switched back to the normal clocksource. But i just might have found the something interesting.. This morning, i wanted to check my mails ( which run in a domU ). I noticed i couldn''t get to it, "Unable to connect", as usual.. But i had cacti open, which could read snmp from the server just fine. I logged in through SSH, and noticed that the date was set to "Fri Dec 22 22:49:54 CET 2226", and not going forward anymore. The clock had stopped. In the meantime, /var/log/messages was spewing messages like this: [134918.952935] clocksource/0: Time went backwards: delta=-6917269600674797899 shadow=134918109266738 offset=843667158 [134923.954592] printk: 1157408 messages suppressed. [134923.954606] clocksource/0: Time went backwards: delta=-6917269595673126078 shadow=134923109294656 offset=845310074 [134928.952308] printk: 1154382 messages suppressed. [134928.952323] clocksource/0: Time went backwards: delta=-6917269590675409336 shadow=134928109321066 offset=842999953 I decided to change something to see if it works, so i did: cat /sys/devices/system/clocksource/clocksource0/current_clocksource, which showed "xen" as usual. I then did echo "jiffies" > /sys/devices/system/clocksource/clocksource0/current_clocksource, which instantly made my clock run again, and stalled programs revived. I saw my webmail ( which was still hanging all the time ) spring to life, along with my previously entered reboot command.. Does this narrow down the search to the solution? Kind regards, Bart ----- Original Message ----- From: "Bart Verwilst" <lists@verwilst.be> To: "Samuel Thibault" <samuel.thibault@ens-lyon.org> Cc: Xen-users@lists.xensource.com Sent: Thursday, October 11, 2007 3:55:28 PM (GMT+0100) Europe/Berlin Subject: Re: [Xen-users] clocksource/0: Time went backwards Another thing i just tried is this: # cat /sys/devices/system/clocksource/clocksource0/current_clocksource xen # cat /sys/devices/system/clocksource/clocksource0/available_clocksource xen jiffies # echo jiffies > /sys/devices/system/clocksource/clocksource0/current_clocksource In dmesg: [57990.836097] Time: jiffies clocksource has been installed. Do you think this might work? Will it break something else? Is this a good idea? Thanks! Bart Verwilst ----- Original Message ----- From: "Samuel Thibault" <samuel.thibault@ens-lyon.org> To: "Bart Verwilst" <lists@verwilst.be> Cc: Xen-users@lists.xensource.com Sent: Thursday, October 11, 2007 3:19:37 PM (GMT+0100) Europe/Berlin Subject: Re: [Xen-users] clocksource/0: Time went backwards Bart Verwilst, le Thu 11 Oct 2007 15:12:19 +0200, a écrit :> Oh.. i was hoping for some simple sysctl tweak or easy patch :) > Bummer :(Oh, actually, the problem you''re getting might be related to CPU frequency changes, try to disable that as well. Samuel _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Bart, hi guys, I''ve to fetch this issue again, because I''m encountering nearly the same problem.> I decided to change something to see if it works, so i did: > cat /sys/devices/system/clocksource/clocksource0/current_clocksource, which showed "xen" as usual. > I then did > echo "jiffies" > /sys/devices/system/clocksource/clocksource0/current_clocksource, which instantly made my clock run again, and stalled programs revived. I saw my webmail ( which was still hanging all the time ) spring to life, along with my previously entered reboot command..Does this fix the problem in the long term? Where did you set this, on dom0 or on domUs or in all of them? Thanks in advance. Regards, Jan -- Artfiles New Media GmbH | Spaldingstr. 160 b | 20097 Hamburg Tel: 040 - 32 02 72 90 | Fax: 040 - 32 02 72 95 E-Mail: support@artfiles.de | Web: http://www.artfiles.de Geschäftsführer Carsten Bals | Handelsregister Hamburg - HRB 81478 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Based on my experiences so far I''m at a loss as to why "xen" ever appears as a clock source, it just seems to make Xen instances totally unreliable. I always add "clock=jiffies" to my "extra" line in DomU''s config files ... If someone can explain the difference between clocks and why Xen is the default and why it doesn''t work I would be very grateful ... Gareth. ----- Original Message ----- step 3.: "Jan Marquardt" <jm@artfiles.de> To: "Bart Verwilst" <lists@verwilst.be> Cc: Xen-users@lists.xensource.com Sent: 22 January 2008 10:15:21 o''clock (GMT) Europe/London Subject: Re: [Xen-users] clocksource/0: Time went backwards Hi Bart, hi guys, I''ve to fetch this issue again, because I''m encountering nearly the same problem.> I decided to change something to see if it works, so i did: > cat /sys/devices/system/clocksource/clocksource0/current_clocksource, which showed "xen" as usual. > I then did > echo "jiffies" > /sys/devices/system/clocksource/clocksource0/current_clocksource, which instantly made my clock run again, and stalled programs revived. I saw my webmail ( which was still hanging all the time ) spring to life, along with my previously entered reboot command..Does this fix the problem in the long term? Where did you set this, on dom0 or on domUs or in all of them? Thanks in advance. Regards, Jan -- Artfiles New Media GmbH | Spaldingstr. 160 b | 20097 Hamburg Tel: 040 - 32 02 72 90 | Fax: 040 - 32 02 72 95 E-Mail: support@artfiles.de | Web: http://www.artfiles.de Geschäftsführer Carsten Bals | Handelsregister Hamburg - HRB 81478 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, does anyone know if this issue has been fixed under Ubuntu Hardy with kernel version 2.6.24 and Xen 3.2? I''m asking this, because maybe we are experiencing some side-effects by setting the clocksource to jiffies. Regards, Jan Gareth Bult wrote:> Hi, > > Based on my experiences so far I''m at a loss as to why "xen" ever appears as a clock source, it just seems to make Xen instances totally unreliable. > > I always add "clock=jiffies" to my "extra" line in DomU''s config files ... > > If someone can explain the difference between clocks and why Xen is the default and why it doesn''t work I would be very grateful ... > > Gareth. > > ----- Original Message ----- > step 3.: "Jan Marquardt" <jm@artfiles.de> > To: "Bart Verwilst" <lists@verwilst.be> > Cc: Xen-users@lists.xensource.com > Sent: 22 January 2008 10:15:21 o''clock (GMT) Europe/London > Subject: Re: [Xen-users] clocksource/0: Time went backwards > > Hi Bart, hi guys, > > I''ve to fetch this issue again, because I''m encountering nearly the same > problem. > >> I decided to change something to see if it works, so i did: >> cat /sys/devices/system/clocksource/clocksource0/current_clocksource, which showed "xen" as usual. >> I then did >> echo "jiffies" > /sys/devices/system/clocksource/clocksource0/current_clocksource, which instantly made my clock run again, and stalled programs revived. I saw my webmail ( which was still hanging all the time ) spring to life, along with my previously entered reboot command.. > > Does this fix the problem in the long term? Where did you set this, on > dom0 or on domUs or in all of them? > > Thanks in advance. > > Regards, > > Jan >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users