Can anyone tell me if the value returned by gettimeofday() on a Xen (Linux) HVM is the true System time or the Domain-virtual time? PS: Domain virtual time is defined as the time progresses at the same pace as cycle counter time, but only while a domain is executing. It stops while the domain is de-scheduled. I am facing issues because my HVMs show a time drift. Thanks _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Sorry for multiple emails. I sent the last one from the wrong address. Can anyone please tell me if the value returned by a time querying instruction like gettimeofday() on a Xen (Linux) HVM is the true (System) time or the Domain-virtual time? PS: Domain virtual time is defined as the time that progresses at the same pace as cycle counter, but only while a domain is executing. It stops while the domain is de-scheduled where as System time accurately reflects the passage of real time. I am facing issues because my HVMs show a time drift. Thanks _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Should be "true" system time, i.e. should be very close to what you see on a "wallclock" (clock on the wall). HVM''s are sadly very widely varied in the parameters needed to minimize time drift. In general in the past, timer_mode=0 (or timer_mode unspecified) would be best for 32-bit Linux domains, timer_mode=1 would be best for Windows domains, and timer_mode=2 would be best for 64-bit Linux domains. However, for best results on Linux, this must be combined with kernel boot parameters that properly select a clock -- and on some Linux kernel versions, the parameters needed are different between 32-bit and 64-bit versions of the same kernel version. It is up to providers of HVM templates (aka "appliances") to choose parameters wisely. Also, you haven''t specified your Xen version, but I believe Xen 4.0 switches the timer_mode default from 0 to 1 so, sadly, clock behavior may change when moving an unchanged HVM domain from pre-4.0 to 4.0. So for best results you should run ntpd in any Linux HVM domain (and I don''t know what you do in Windows). But even ntpd may be inadequate to avoid drift if poor parameters are chosen. ========= From: Priya [mailto:pbhat@acis.ufl.edu] Sent: Thursday, February 25, 2010 9:04 AM To: xen-devel@lists.xensource.com; xen-users@lists.xensource.com Subject: [Xen-devel] Domain-Virtual time Sorry for multiple emails. I sent the last one from the wrong address. Can anyone please tell me if the value returned by a time querying instruction like gettimeofday() on a Xen (Linux) HVM is the true (System) time or the Domain-virtual time? PS: Domain virtual time is defined as the time that progresses at the same pace as cycle counter, but only while a domain is executing. It stops while the domain is de-scheduled where as System time accurately reflects the passage of real time. I am facing issues because my HVMs show a time drift. Thanks _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thanks Dan! That''s a lot of useful information. Since yesterday, I started NTP on my machines to correct and measure the amount of drift. (I am running 3 HVMs with a Ubuntu 8.04 Linux (tick-less) kernel which were all installed in an identical manner. At the time of installing the HVMs I did not change the default timer_mode). The funny thing is that NTP is measuring a very different drift on my three machines (-189.206, -108.373 and -71.321 parts per million). The drift reported on Domain-0 is -11.393. So I don''t think my machines are showing the system time. In addition, the negative sign on the drift means that my machines are running *faster* that the real time, which is again puzzling. I found out that VMWare has issues with overcompensation on its linux kernels that cause the VM time to run faster. Could Xen be having a similar problem ? The fact that all three on my machines are showing different drifts makes me doubt that they are showing the system time. I checked the CPU frequency that the three machines are reporting (from /proc/cpuinfo) and they are similar but not identical. Any thoughts? On Thu, Feb 25, 2010 at 3:36 PM, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:> Should be "true" system time, i.e. should be very close to what > you see on a "wallclock" (clock on the wall). > > HVM''s are sadly very widely varied in the parameters needed > to minimize time drift. In general in the past, timer_mode=0 > (or timer_mode unspecified) would be best for 32-bit Linux > domains, timer_mode=1 would be best for Windows domains, > and timer_mode=2 would be best for 64-bit Linux domains. > However, for best results on Linux, this must be combined with > kernel boot parameters that properly select a clock -- and > on some Linux kernel versions, the parameters needed are > different between 32-bit and 64-bit versions of the same > kernel version. It is up to providers of HVM templates > (aka "appliances") to choose parameters wisely. > > Also, you haven''t specified your Xen version, but I believe > Xen 4.0 switches the timer_mode default from 0 to 1 so, sadly, > clock behavior may change when moving an unchanged HVM > domain from pre-4.0 to 4.0. > > So for best results you should run ntpd in any Linux HVM > domain (and I don''t know what you do in Windows). But > even ntpd may be inadequate to avoid drift if poor parameters > are chosen. > > =========> > From: Priya [mailto:pbhat@acis.ufl.edu] > Sent: Thursday, February 25, 2010 9:04 AM > To: xen-devel@lists.xensource.com; xen-users@lists.xensource.com > Subject: [Xen-devel] Domain-Virtual time > > Sorry for multiple emails. I sent the last one from the wrong address. > > Can anyone please tell me if the value returned by a time querying > instruction like gettimeofday() on a Xen (Linux) HVM is the true (System) > time or the Domain-virtual time? > > PS: Domain virtual time is defined as the time that progresses at the same > pace as cycle counter, but only while a domain is executing. It stops while > the domain is de-scheduled where as System time accurately reflects the > passage of real time. > > I am facing issues because my HVMs show a time drift. > > Thanks > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, At 16:46 +0000 on 26 Feb (1267202798), Priya wrote:> The funny thing is that NTP is measuring a very different drift on my > three machines (-189.206, -108.373 and -71.321 parts per million). The > drift reported on Domain-0 is -11.393. So I don''t think my machines > are showing the system time.No; they are showing system time modified according to the timer_mode setting, then extrapolated into a collection of virtual timers and reconstituted by the linux kernel. I don''t know a lot about linux HVM because I always run Linux with PV kernels, but on Windows I''ve found that the (virtual) ACPI PM timer is a better time-source than the HPET or RTC.> In addition, the negative sign on the drift means that my machines are > running faster that the real time, which is again puzzling.Xen system time generally does drift forward, IIRC that''s because xen always tends to catch up to the fastest CPU, so cross-CPU jitter turns into forward drift. Linux may be doing the same thing inside the HVM VM, where the jitter is higher. Tim. -- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, XenServer Engineering Citrix Systems UK Ltd. (Company #02937203, SL9 0BG) _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thanks Tim! I have some follow-up questions: 1. How can I check/change the timer_mode on a *XenServer* HVM and on a Xen HVM? 2. How can I check/change the time-source on my machines? 3.> > No; they are showing system time modified according to the timer_mode > setting, then extrapolated into a collection of virtual timers and > reconstituted by the linux kernel.Can you please elaborate a little bit. Or please point me to a reference. Thanks again ! Pr On Fri, Feb 26, 2010 at 12:01 PM, Tim Deegan <Tim.Deegan@citrix.com> wrote:> Hi, > > At 16:46 +0000 on 26 Feb (1267202798), Priya wrote: > > The funny thing is that NTP is measuring a very different drift on my > > three machines (-189.206, -108.373 and -71.321 parts per million). The > > drift reported on Domain-0 is -11.393. So I don''t think my machines > > are showing the system time. > > No; they are showing system time modified according to the timer_mode > setting, then extrapolated into a collection of virtual timers and > reconstituted by the linux kernel. > > I don''t know a lot about linux HVM because I always run Linux with PV > kernels, but on Windows I''ve found that the (virtual) ACPI PM timer is a > better time-source than the HPET or RTC. > > > In addition, the negative sign on the drift means that my machines are > > running faster that the real time, which is again puzzling. > > Xen system time generally does drift forward, IIRC that''s because xen > always tends to catch up to the fastest CPU, so cross-CPU jitter turns > into forward drift. Linux may be doing the same thing inside the HVM > VM, where the jitter is higher. > > Tim. > > -- > Tim Deegan <Tim.Deegan@citrix.com> > Principal Software Engineer, XenServer Engineering > Citrix Systems UK Ltd. (Company #02937203, SL9 0BG) > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Priya -- You need only run additional domains (thus "overcommitting" the CPU(s) and guaranteeing that NO domain is getting close to an entire CPU) to demonstrate that it is certainly not domain time. It IS system time but, as the VMware paper describes, virtualizing time on an older (buggy) OS is more of an art than a science. Unless one builds directly into the hypervisor a complete list of all OS''s (including all versions and patch-levels) and all of their bugs and idiosyncrasies, the result is a s*it-load of options at different levels of the virtualization stack (including hardware configuration such as in a boot-time BIOS menu). As you may also have read, even on a physical machine running a native OS, time drift is very possible as time in every system is based on one or more inexpensive crystals with frequencies that are only estimates of a (extremely expensive) cesium clock (which for lack of a better term we will call "true" time) and may themselves drift relative to true time and relative to each other on the same system or even an apparently identical system. This can often be demonstrated as you saw by checking /proc/cpuinfo... but identical values don''t mean that the clocks are true, just that the difference is lower than the kernel code measuring it can discern or choose to report. When virtualizing time, some of the problems manifest in time drift in the virtual system which moves faster than true time and some slower than true time. NTP does its best to find a source of true time and then adjusts the clock on the system it is controlling so that it asymptotically approaches true time withOUT allowing time to go backwards (as this can cause all sorts of challenging system problems... imagine a "make" where time randomly goes backwards in the middle!). But NTP may not have a good source for true time (as it is an administrator who configures it), NTP may subtly conflict with some combinations of the options set by administrators in the virtualization stack (because not everyone has access to "true" time... for example suppose your virtual machine has not networking configured), and NTP may silently give up if the drift is too bad for it to safely compensate. (On a physical machine, this would be considered a "hardware bug" and you would ship your box back to the vendor to get one without a "broken" clock.) So if you have ever heard the old pop song by Chicago "Does anybody really know what time it is?", the title is truer than most people believe ;-) Hope that helps! Dan From: Priya [mailto:pbhat@acis.ufl.edu] Sent: Friday, February 26, 2010 9:47 AM To: Dan Magenheimer Cc: xen-devel@lists.xensource.com; xen-users@lists.xensource.com Subject: Re: [Xen-devel] Domain-Virtual time Thanks Dan! That''s a lot of useful information. Since yesterday, I started NTP on my machines to correct and measure the amount of drift. (I am running 3 HVMs with a Ubuntu 8.04 Linux (tick-less) kernel which were all installed in an identical manner. At the time of installing the HVMs I did not change the default timer_mode). The funny thing is that NTP is measuring a very different drift on my three machines (-189.206, -108.373 and -71.321 parts per million). The drift reported on Domain-0 is -11.393. So I don''t think my machines are showing the system time. In addition, the negative sign on the drift means that my machines are running faster that the real time, which is again puzzling. I found out that VMWare has issues with overcompensation on its linux kernels that cause the VM time to run faster. Could Xen be having a similar problem ? The fact that all three on my machines are showing different drifts makes me doubt that they are showing the system time. I checked the CPU frequency that the three machines are reporting (from /proc/cpuinfo) and they are similar but not identical. Any thoughts? On Thu, Feb 25, 2010 at 3:36 PM, Dan Magenheimer <dan.magenheimer@oracle.com> wrote: Should be "true" system time, i.e. should be very close to what you see on a "wallclock" (clock on the wall). HVM''s are sadly very widely varied in the parameters needed to minimize time drift. In general in the past, timer_mode=0 (or timer_mode unspecified) would be best for 32-bit Linux domains, timer_mode=1 would be best for Windows domains, and timer_mode=2 would be best for 64-bit Linux domains. However, for best results on Linux, this must be combined with kernel boot parameters that properly select a clock -- and on some Linux kernel versions, the parameters needed are different between 32-bit and 64-bit versions of the same kernel version. It is up to providers of HVM templates (aka "appliances") to choose parameters wisely. Also, you haven''t specified your Xen version, but I believe Xen 4.0 switches the timer_mode default from 0 to 1 so, sadly, clock behavior may change when moving an unchanged HVM domain from pre-4.0 to 4.0. So for best results you should run ntpd in any Linux HVM domain (and I don''t know what you do in Windows). But even ntpd may be inadequate to avoid drift if poor parameters are chosen. ========= From: Priya [mailto:pbhat@acis.ufl.edu] Sent: Thursday, February 25, 2010 9:04 AM To: xen-devel@lists.xensource.com; xen-users@lists.xensource.com Subject: [Xen-devel] Domain-Virtual time Sorry for multiple emails. I sent the last one from the wrong address. Can anyone please tell me if the value returned by a time querying instruction like gettimeofday() on a Xen (Linux) HVM is the true (System) time or the Domain-virtual time? PS: Domain virtual time is defined as the time that progresses at the same pace as cycle counter, but only while a domain is executing. It stops while the domain is de-scheduled where as System time accurately reflects the passage of real time. I am facing issues because my HVMs show a time drift. Thanks _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
The timer_mode=X can be set in the "VM configuration" file in which you specify things like the disk, memory, etc. Setting the Xen time source on your machine is done by adding clocksource=xxxxx to your Xen boot line, but unless your system has one broken clocksource and another is fine, this is rarely the source of extreme drift in an HVM. (And, by the way, doing something like enabling HPET in your BIOS may cause Xen to choose a different clocksource. Xen tries to choose what it deems to be the best, and the clocksource= is just a manual override.) There are similar Linux kernel parameters that must be specified within the boot-time grub file for the HVM. These parameters vary from release to release (and between 32-bit and 64-bit). Many more recent kernels allow you to change the clocksource dynamically using a sysfs file. Be aware that in some cases, a misspelled option name or one that works fine on one version but not on the currently booting Xen/kernel, may be completely and silently ignored. And finally, time has been a moving target on Xen, Linux, VMware, etc, so problems that appear in older versions may have been resolved in later versions. ANY time you post questions/problems regarding time (or for that matter even non-time-related problems), it is best to describe what version of Xen and kernels you are using. ===== From: Priya [mailto:pbhat@acis.ufl.edu] Sent: Friday, February 26, 2010 10:20 AM To: Tim Deegan Cc: Dan Magenheimer; xen-devel@lists.xensource.com; xen-users@lists.xensource.com Subject: Re: [Xen-devel] Domain-Virtual time Thanks Tim! I have some follow-up questions: 1. How can I check/change the timer_mode on a XenServer HVM and on a Xen HVM? 2. How can I check/change the time-source on my machines? 3. No; they are showing system time modified according to the timer_mode setting, then extrapolated into a collection of virtual timers and reconstituted by the linux kernel. Can you please elaborate a little bit. Or please point me to a reference. Thanks again ! Pr On Fri, Feb 26, 2010 at 12:01 PM, Tim Deegan <Tim.Deegan@citrix.com> wrote: Hi, At 16:46 +0000 on 26 Feb (1267202798), Priya wrote:> The funny thing is that NTP is measuring a very different drift on my > three machines (-189.206, -108.373 and -71.321 parts per million). The > drift reported on Domain-0 is -11.393. So I don''t think my machines > are showing the system time.No; they are showing system time modified according to the timer_mode setting, then extrapolated into a collection of virtual timers and reconstituted by the linux kernel. I don''t know a lot about linux HVM because I always run Linux with PV kernels, but on Windows I''ve found that the (virtual) ACPI PM timer is a better time-source than the HPET or RTC.> In addition, the negative sign on the drift means that my machines are > running faster that the real time, which is again puzzling.Xen system time generally does drift forward, IIRC that''s because xen always tends to catch up to the fastest CPU, so cross-CPU jitter turns into forward drift. Linux may be doing the same thing inside the HVM VM, where the jitter is higher. Tim. -- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, XenServer Engineering Citrix Systems UK Ltd. (Company #02937203, SL9 0BG) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks guys ! I am really grateful ! On Fri, Feb 26, 2010 at 12:46 PM, Dan Magenheimer < dan.magenheimer@oracle.com> wrote:> The timer_mode=X can be set in the "VM configuration" file > in which you specify things like the disk, memory, etc. > > Setting the Xen time source on your machine is done by > adding clocksource=xxxxx to your Xen boot line, but unless > your system has one broken clocksource and another is fine, > this is rarely the source of extreme drift in an HVM. > (And, by the way, doing something like enabling HPET in > your BIOS may cause Xen to choose a different clocksource. > Xen tries to choose what it deems to be the best, and > the clocksource= is just a manual override.) > > There are similar Linux kernel parameters that must > be specified within the boot-time grub file for the HVM. > These parameters vary from release to release (and > between 32-bit and 64-bit). Many more recent kernels > allow you to change the clocksource dynamically > using a sysfs file. > > Be aware that in some cases, a misspelled option name > or one that works fine on one version but not on the > currently booting Xen/kernel, may be completely and silently > ignored. > > And finally, time has been a moving target on Xen, > Linux, VMware, etc, so problems that appear in older > versions may have been resolved in later versions. > ANY time you post questions/problems regarding time > (or for that matter even non-time-related problems), > it is best to describe what version of Xen and kernels > you are using. > > =====> > From: Priya [mailto:pbhat@acis.ufl.edu] > Sent: Friday, February 26, 2010 10:20 AM > To: Tim Deegan > Cc: Dan Magenheimer; xen-devel@lists.xensource.com; > xen-users@lists.xensource.com > Subject: Re: [Xen-devel] Domain-Virtual time > > Thanks Tim! > > I have some follow-up questions: > 1. How can I check/change the timer_mode on a XenServer HVM and on a Xen > HVM? > 2. How can I check/change the time-source on my machines? > 3. No; they are showing system time modified according to the timer_mode > setting, then extrapolated into a collection of virtual timers and > reconstituted by the linux kernel. > Can you please elaborate a little bit. Or please point me to a reference. > Thanks again ! > Pr > > > On Fri, Feb 26, 2010 at 12:01 PM, Tim Deegan <Tim.Deegan@citrix.com> > wrote: > Hi, > > At 16:46 +0000 on 26 Feb (1267202798), Priya wrote: > > The funny thing is that NTP is measuring a very different drift on my > > three machines (-189.206, -108.373 and -71.321 parts per million). The > > drift reported on Domain-0 is -11.393. So I don''t think my machines > > are showing the system time. > No; they are showing system time modified according to the timer_mode > setting, then extrapolated into a collection of virtual timers and > reconstituted by the linux kernel. > > I don''t know a lot about linux HVM because I always run Linux with PV > kernels, but on Windows I''ve found that the (virtual) ACPI PM timer is a > better time-source than the HPET or RTC. > > > In addition, the negative sign on the drift means that my machines are > > running faster that the real time, which is again puzzling. > Xen system time generally does drift forward, IIRC that''s because xen > always tends to catch up to the fastest CPU, so cross-CPU jitter turns > into forward drift. Linux may be doing the same thing inside the HVM > VM, where the jitter is higher. > > Tim. > > -- > Tim Deegan <Tim.Deegan@citrix.com> > Principal Software Engineer, XenServer Engineering > Citrix Systems UK Ltd. (Company #02937203, SL9 0BG) > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel