Hello! I have understood that when /proc/sys/xen/independent_wallclock is 0, then domU clock stays in sync with dom0 clock. Now I have a case with Xen 3.1.2 (also occured with 3.1.1, but I haven''t documented it) where domU clock runs significantly faster - after 65 hours domU clock is 20 seconds ahead of dom0 clock. In this case dom0 is not running ntpd. When ntpd is running on dom0, domU clock also stays correct. Is this normal behaviour? Some examples to show clock differences: domU# uptime 14:54:09 up 2 days, 17:05, 1 user, load average: 0.75, 0.49, 0.43 domU# ntpdate -q ntp.eenet.ee server 193.40.133.142, stratum 1, offset -20.856732, delay 0.02579 19 Nov 14:54:13 ntpdate[8467]: step time server 193.40.133.142 offset -20.856732 sec dom0# uptime 14:55:02 up 2 days, 17:06, 1 user, load average: 0.07, 0.02, 0.00 dom0# ntpdate -q ntp.eenet.ee server 193.40.133.142, stratum 1, offset -0.438055, delay 0.02589 19 Nov 14:55:17 ntpdate[32284]: adjust time server 193.40.133.142 offset -0.438055 sec Xen version used: Xen version 3.1.2 (jussuf@localdomain) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) Fri Nov 16 17:52:59 EET 2007 Latest ChangeSet: Wed Nov 14 23:35:43 2007 +0000 15502:c6776b6da8ee Linux xen-2 2.6.18.8-xen0 #1 SMP Fri Nov 16 17:39:31 EET 2007 x86_64 GNU/Linux -- Tõnu Raitviir tonu.raitviir@eenet.ee EENet http://www.eenet.ee/ BalticGrid Project http://www.balticgrid.org/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Clock skew is a known problem in virtualized systems. VMWare wrote a whitepaper a while back which goes into some of the problems that they faced while dealing with this same issue... http://www.vmware.com/pdf/vmware_timekeeping.pdf The Xen project is not immune to these problems - however, some efforts have been made recently on the list to address some of these issues See these threads: http://lists.xensource.com/archives/html/xen-devel/2007-10/msg01010.html http://lists.xensource.com/archives/html/xen-devel/2007-11/msg00554.html What it basically comes down to is that it is necessary to keep the clock skew under 0.05% for NTP to work - so that was the design goal. The code introduced by some recent patches claims to be accurate to 0.02% (for at least HVM guests) Hope this helps. Ben Tõnu Raitviir wrote:> > > Hello! > I have understood that when /proc/sys/xen/independent_wallclock is 0, > then domU clock stays in sync with dom0 clock. Now I have a case with > Xen 3.1.2 (also occured with 3.1.1, but I haven''t documented it) where > domU clock runs significantly faster - after 65 hours domU clock is 20 > seconds ahead of dom0 clock. > In this case dom0 is not running ntpd. When ntpd is running on dom0, > domU clock also stays correct. Is this normal behaviour? > > Some examples to show clock differences: > > domU# uptime > 14:54:09 up 2 days, 17:05, 1 user, load average: 0.75, 0.49, 0.43 > > domU# ntpdate -q ntp.eenet.ee > server 193.40.133.142, stratum 1, offset -20.856732, delay 0.02579 > 19 Nov 14:54:13 ntpdate[8467]: step time server 193.40.133.142 offset > -20.856732 sec > > dom0# uptime > 14:55:02 up 2 days, 17:06, 1 user, load average: 0.07, 0.02, 0.00 > > dom0# ntpdate -q ntp.eenet.ee > server 193.40.133.142, stratum 1, offset -0.438055, delay 0.02589 > 19 Nov 14:55:17 ntpdate[32284]: adjust time server 193.40.133.142 > offset -0.438055 sec > > > Xen version used: > > Xen version 3.1.2 (jussuf@localdomain) (gcc version 4.1.2 20061115 > (prerelease) (Debian 4.1.1-21)) Fri Nov 16 17:52:59 EET 2007 > Latest ChangeSet: Wed Nov 14 23:35:43 2007 +0000 15502:c6776b6da8ee > > Linux xen-2 2.6.18.8-xen0 #1 SMP Fri Nov 16 17:39:31 EET 2007 x86_64 > GNU/Linux > > ------------------------------------------------------------------------ > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Do all domUs stay in sync with each other, or do all guests drift randomly? -- Keir On 20/11/07 13:33, "Tõnu Raitviir" <jussuf@eenet.ee> wrote:> > > Hello! > I have understood that when /proc/sys/xen/independent_wallclock is 0, then > domU clock stays in sync with dom0 clock. Now I have a case with Xen 3.1.2 > (also occured with 3.1.1, but I haven''t documented it) where domU clock > runs significantly faster - after 65 hours domU clock is 20 seconds ahead > of dom0 clock. > In this case dom0 is not running ntpd. When ntpd is running on dom0, domU > clock also stays correct. Is this normal behaviour? > > Some examples to show clock differences: > > domU# uptime > 14:54:09 up 2 days, 17:05, 1 user, load average: 0.75, 0.49, 0.43 > > domU# ntpdate -q ntp.eenet.ee > server 193.40.133.142, stratum 1, offset -20.856732, delay 0.02579 > 19 Nov 14:54:13 ntpdate[8467]: step time server 193.40.133.142 offset > -20.856732 sec > > dom0# uptime > 14:55:02 up 2 days, 17:06, 1 user, load average: 0.07, 0.02, 0.00 > > dom0# ntpdate -q ntp.eenet.ee > server 193.40.133.142, stratum 1, offset -0.438055, delay 0.02589 > 19 Nov 14:55:17 ntpdate[32284]: adjust time server 193.40.133.142 offset > -0.438055 sec > > > Xen version used: > > Xen version 3.1.2 (jussuf@localdomain) (gcc version 4.1.2 20061115 > (prerelease) (Debian 4.1.1-21)) Fri Nov 16 17:52:59 EET 2007 > Latest ChangeSet: Wed Nov 14 23:35:43 2007 +0000 15502:c6776b6da8ee > > Linux xen-2 2.6.18.8-xen0 #1 SMP Fri Nov 16 17:39:31 EET 2007 x86_64 GNU/Linux_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, 20 Nov 2007, Keir Fraser wrote:> Do all domUs stay in sync with each other, or do all guests drift randomly?I didn''t compare domU-s before, but they seem to be in sync. I stopped ntpd on dom0 five hours ago, and now the results of ntpdate are: dom0 offset -0.002353 sec domU #1 offset -1.555219 sec domU #2 offset -1.555261 sec domU #3 offset -1.555243 sec -- Tõnu Raitviir tonu.raitviir@eenet.ee EENet http://www.eenet.ee/ BalticGrid Project http://www.balticgrid.org/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I wonder if dom0 has remembered some ntp-derived adjustment values despite ntpd no longer running. That would explain why it continues to keep good time while all others drift. The others drifting is quite explicable -- they are only as accurate as our estimate of platform timer (PIT or HPET) frequency at boot time, and the accuracy of the crystal that drives that timer is probably only good to about 100ppm. So... running ntpd in dom0 is a good idea. :-) -- Keir On 20/11/07 19:15, "Tõnu Raitviir" <jussuf@eenet.ee> wrote:> On Tue, 20 Nov 2007, Keir Fraser wrote: > >> Do all domUs stay in sync with each other, or do all guests drift randomly? > > I didn''t compare domU-s before, but they seem to be in sync. I stopped > ntpd on dom0 five hours ago, and now the results of ntpdate are: > dom0 offset -0.002353 sec > domU #1 offset -1.555219 sec > domU #2 offset -1.555261 sec > domU #3 offset -1.555243 sec_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, 20 Nov 2007, Keir Fraser wrote:> I wonder if dom0 has remembered some ntp-derived adjustment values despite > ntpd no longer running. That would explain why it continues to keep good > time while all others drift.Well... in the example in my first letter dom0 was booted without ntpd and after 2+ days its clock offset was only -0.438055 while domU had drifted far away. In that case there was no adjustment values to remember.> So... running ntpd in dom0 is a good idea. :-)I don''t question that. But I wonder, isn''t domU clock supposed to be always in sync with dom0, with or without ntpd? Or how exactly does ntpd in dom0 affect domU clock? -- Tõnu Raitviir tonu.raitviir@eenet.ee EENet http://www.eenet.ee/ BalticGrid Project http://www.balticgrid.org/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 20/11/07 21:52, "Tõnu Raitviir" <jussuf@eenet.ee> wrote:>> I wonder if dom0 has remembered some ntp-derived adjustment values despite >> ntpd no longer running. That would explain why it continues to keep good >> time while all others drift. > > Well... in the example in my first letter dom0 was booted without ntpd and > after 2+ days its clock offset was only -0.438055 while domU had drifted > far away. In that case there was no adjustment values to remember.Could it not be a drift? e.g., Perhaps you ran ntpdate in dom0 but not the domUs?>> So... running ntpd in dom0 is a good idea. :-) > > I don''t question that. But I wonder, isn''t domU clock supposed to be > always in sync with dom0, with or without ntpd? Or how exactly does ntpd > in dom0 affect domU clock?dom0 updates Xen''s shared notion of wallclock time periodically, but only if dom0 is locked onto ntp. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, 20 Nov 2007, Keir Fraser wrote:> Could it not be a drift? e.g., Perhaps you ran ntpdate in dom0 but not the > domUs?That''s possible, probably ntpdate was executed automatically on dom0. Now I did a new test, rebooted dom0 and made sure that no ntp* commands are run. Results are somewhat different: # uptime 14:21:41 up 12:31, 1 user, load average: 0.00, 0.00, 0.00 dom0 offset -7.190458 domU #1 offset -4.380381 domU #2 offset -4.380430 domU #3 offset -4.380401 So without any ntp interaction dom0 clock is running faster than domU clock and all domU clocks are in sync with each other, but still not with dom0. Now I''m going to start ntpd again and make sure that it never stops :-) -- Tõnu Raitviir tonu.raitviir@eenet.ee EENet http://www.eenet.ee/ BalticGrid Project http://www.balticgrid.org/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel