hello, I''m running a 2.0 snapshot from sept 11 with 2.6.8.1, and the system time is jumping back and forth in a domain -- partly it is synchronized with Dom0''s time (which is set via ntpd), but then it jumps ahead, and then back again, as seen in these consecutive lines from syslog: Sep 17 09:22:39 [...] Sep 17 10:34:24 [...] Sep 17 09:23:40 [...] 9:22/9:23 is the correct time. I''ve got the impression (but have not analyzed it systematically) that a few minutes after boot, everything turns normal. the domain is not running ntpd or anything else which would meddle with the clock. machine is a dual PII-450. any ideas on how to debug this further? cm. -- ** christian mock in vienna, austria -- http://www.tahina.priv.at/~cm/> Konzept Kunde/König?Ist deren Internet kaputt oder meines? Kann ich das wieder hinbiegen oder die? -- Ralph Angenendt in dasr ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> I''m running a 2.0 snapshot from sept 11 with 2.6.8.1, and the system > time is jumping back and forth in a domain -- partly it is > synchronized with Dom0''s time (which is set via ntpd), but then it > jumps ahead, and then back again, as seen in these consecutive lines > from syslog: > > Sep 17 09:22:39 [...] > Sep 17 10:34:24 [...] > Sep 17 09:23:40 [...] > > 9:22/9:23 is the correct time. I''ve got the impression (but have not > analyzed it systematically) that a few minutes after boot, everything > turns normal.Very odd. Presumably you''re running your un-priv domains with /proc/sys/xen/independent_wallclock set to the default of 0? In this event, any attempt to set or adjust the wall clock time in an unpriv domain should be ignored: it should remain slaved to dom0''s wall clock time, which will either be free running from the hardware RTC, or synchronised via NTP if you''re running ntpd. This certainly all seemed to work fine for 2.4. Are you 100% sure you haven''t got ntpdate running in your startup scripts or crontab? It should be ignored, but its possible that the port from 2.4 to 2.6 wasn''t quite right. Ian ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Mon, Sep 20, 2004 at 10:50:28PM +0100, Ian Pratt wrote:> Presumably you''re running your un-priv domains with > /proc/sys/xen/independent_wallclock set to the default of 0?Yes.> Are you 100% sure you haven''t got ntpdate running in your startup > scripts or crontab? It should be ignored, but its possible that > the port from 2.4 to 2.6 wasn''t quite right.Yup. There''s nothing in the running processes that should try to mess with the clock; in the init scripts, the only thing that does try is hwclock, and that bails out because it tries to to I/O access and can''t -- and even if it could, it''s a one-time thing, and I''m seeing jumps long after it tries to run. Now that I''ve written a few lines of perl to report time jumps, I don''t see them. Heisenbug. I did, however, run a script over Domain0''s syslog files to verify time continuity there, and it seems Dom0 is not affected. cm. -- ** christian mock in vienna, austria -- http://www.tahina.priv.at/~cm/ "Tauschboerse, die. Neudeutsch im Usenetjargon fuer Flamewar (Heftiger Austausch von Beleidigungen und verbalen Ohrfeigen)." -- at in aip ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> > Are you 100% sure you haven''t got ntpdate running in your startup > > scripts or crontab? It should be ignored, but its possible that > > the port from 2.4 to 2.6 wasn''t quite right. > > Yup. There''s nothing in the running processes that should try to mess > with the clock; in the init scripts, the only thing that does try is > hwclock, and that bails out because it tries to to I/O access and > can''t -- and even if it could, it''s a one-time thing, and I''m seeing > jumps long after it tries to run. > > Now that I''ve written a few lines of perl to report time jumps, I > don''t see them. Heisenbug.I''ve got a little program that does successive calls to gettimeofday to look for such problems, and I haven''t seen any such issues in quite a while. I presume syslogd uses gettimeofday to figure out what the time is? It might be worth strace''ing it to see if you can catch things going wrong. Ian ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Wed, 22 Sep 2004, christian mock wrote:> Now that I''ve written a few lines of perl to report time jumps, I > don''t see them. Heisenbug.Hah, that''s my experience too. I tried to reproduce the bug for a few hours today and nothing happened ;) -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
I am have been seeing xen0 time jump around all weekend. I have not been able to pin down a way to reproduce the problem. It will appear like the clocks are drifting unlocked from ntp and then suddenly most are right on time. A while later some clocks are off an hour and some are fine. Sometimes checking or setting hwclock makes thing jump. Some hosts seem more prone to it than others. This is with the today''s pull, changeset 1.1347 and running ntpd 4.2.0a I''m wondering if ntpd is having trouble synchronizing to its server. David ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> > I am have been seeing xen0 time jump around all weekend. I have not been > able to pin down a way to reproduce the problem. > > It will appear like the clocks are drifting unlocked from ntp and then > suddenly most are right on time. A while later some clocks are off an > hour and some are fine. Sometimes checking or setting hwclock makes > thing jump. Some hosts seem more prone to it than others.Are you 100% sure it worked before? The only time related change is 1.1338.1.1 "Leave the TSC cpu feature bit set. ", but I''d be very surprised if this changed the behaviour of NTP.> This is with the today''s pull, changeset 1.1347 and running ntpd 4.2.0a > I''m wondering if ntpd is having trouble synchronizing to its server.ntpq then ''peers'' should tell you this. I wouldn''t expect to seem time jumping by an hour, though. Hmm, I wander if its something daft to do with hwclock being UTC or local time? Are you running anything which is trying to set the hwclock in dom0? It''s best to leave well alone and let Xen do this. Ian ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> > I am have been seeing xen0 time jump around all weekend. I have not been > able to pin down a way to reproduce the problem. > > It will appear like the clocks are drifting unlocked from ntp and then > suddenly most are right on time. A while later some clocks are off an > hour and some are fine. Sometimes checking or setting hwclock makes > thing jump. Some hosts seem more prone to it than others. > > This is with the today''s pull, changeset 1.1347 and running ntpd 4.2.0a > I''m wondering if ntpd is having trouble synchronizing to its server.Sounds possible. I think the problem is that the kernel''s STA_UNSYNC flag is changing, and so DOM0 is sometimes pulling wallclock time from Xen and sometimes from the hardware clock or an NTP server. If any of these is wildly out (e.g., the CMOS clock, by an hour because of daylight saving) then time will be very screwed. I''ve checked in a fix which stops DOM0 from ever pulling time from Xen -- after boot time, no good can ever come of it as Xen''s timebase is no more accurate than DOM0''s. More generally -- if anyone sees problems with time jumping in DOM >0, it may be because you are running ntpd in that domain. It is only necessary to run ntpd in DOM0, but if you *must* run ntpd in others then add ''independent_wallclock'' to the kernel command line when you boot them. -- Keir ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
" Are you running anything which is trying to set the hwclock in " dom0? It''s best to leave well alone and let Xen do this. At shutdown the xen0 does try to set the hwclock. Right after stopping ntpd, debian tries to save the system time to the hardware clock. With yesterday''s time fix, 7 of 16 hosts hang trying to shutdown. The one hung host on which I have a console is trying write the hardware clock. From previoius experience, I suspect if left alone an hour or so they will complete their shutdown and reboot. ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
My solution was to simply comment out the hwclock program where ever I found it. ;) On Tue, 2004-09-28 at 08:29, David Becker wrote:> " Are you running anything which is trying to set the hwclock in > " dom0? It''s best to leave well alone and let Xen do this. > > At shutdown the xen0 does try to set the hwclock. Right after stopping > ntpd, debian tries to save the system time to the hardware clock. > > With yesterday''s time fix, 7 of 16 hosts hang trying to shutdown. > The one hung host on which I have a console is trying write > the hardware clock. From previoius experience, I suspect if left alone > an hour or so they will complete their shutdown and reboot. > > > > ------------------------------------------------------- > This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 > Project Admins to receive an Apple iPod Mini FREE for your judgement on > who ports your project to Linux PPC the best. Sponsored by IBM. > Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xen-devel------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel