Xen users - Jan 2013 - Ever increasing time offset for HVM domain / Huge amounts of drift

Hi,

I am currently running Xen 4.2.1 (this has also happened in 4.2.0 as well).  We
have been having a major problem with sometimes huge amounts of clock drift in
Windows VMs.  Sometimes the clock on a VM could suddenly jump by over a week
(usually forwards, however time has been known to go backwards as well).

Now I don''t profess to know the internals of Xen, however through my
investigation I believe I have a degree of knowledge of what could be causing
the problem.

The steps to reproduce this (for me at least), is to simply do a manual NTP sync
on a Windows VM.  Upon monitoring the qemu-dm log file for the VM, I see similar
to the following:

Time offset set 489, added offset 480
Time offset set 436, added offset -53
Time offset set 496, added offset 60
Time offset set 494, added offset -2
Time offset set 554, added offset 60
Time offset set 565, added offset 11
Time offset set 606, added offset 41
Time offset set -1974, added offset -2580
Time offset set 1626, added offset 3600
Time offset set 1579, added offset -47
Time offset set 1639, added offset 60

It seems to add the same number of seconds to the offset as has passed since the
last sync.  The offset just keeps on increasing, eventually resulting in huge
numbers equating to days.  Occasionally the offset may jump a bit and go down
but the general trend is up.  Although this does not affect the VM immediately,
at some point I am guessing it syncs itself with the CMOS clock (which is now a
large number of seconds offset from the actual time), resulting in a huge jump
in time.  A reboot is a guaranteed way to get the new, incorrect time.

Although I do not understand all of the underlying code, I presume the correct
way this should work is it should be comparing the CMOS time that''s
just been set with the hardware clock on the physical machine, resulting in an
offset between the two.  This would result in a generally stable number (ideally
0).  Obviously it is incorrect behaviour for the number to keep going up.  To my
mind it looks like it may be somehow getting an inaccurate time from the system
(in many cases a fixed time rather than an up-to-date current time).

Does anyone have any light they may be able to shed on this?  Is it possible it
could be struggling to get an accurate time from the hardware?  I have checked
on several occasions and both the system time and the BIOS clock are spot on.

Regards,
Phil.



_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

Who knows the reason?


----- Original Message ----- 
From: Phil Evans
To: xen-users@lists.xen.org
Sent: Friday, January 11, 2013 1:01 AM
Subject: [Xen-users] Ever increasing time offset for HVM domain / 
Hugeamounts of drift


Hi,

I am currently running Xen 4.2.1 (this has also happened in 4.2.0 as well). 
We have been having a major problem with sometimes huge amounts of clock 
drift in Windows VMs.  Sometimes the clock on a VM could suddenly jump by 
over a week (usually forwards, however time has been known to go backwards 
as well).

Now I don''t profess to know the internals of Xen, however through my 
investigation I believe I have a degree of knowledge of what could be 
causing the problem.

The steps to reproduce this (for me at least), is to simply do a manual NTP 
sync on a Windows VM.  Upon monitoring the qemu-dm log file for the VM, I 
see similar to the following:

Time offset set 489, added offset 480
Time offset set 436, added offset -53
Time offset set 496, added offset 60
Time offset set 494, added offset -2
Time offset set 554, added offset 60
Time offset set 565, added offset 11
Time offset set 606, added offset 41
Time offset set -1974, added offset -2580
Time offset set 1626, added offset 3600
Time offset set 1579, added offset -47
Time offset set 1639, added offset 60

It seems to add the same number of seconds to the offset as has passed since 
the last sync.  The offset just keeps on increasing, eventually resulting in 
huge numbers equating to days.  Occasionally the offset may jump a bit and 
go down but the general trend is up.  Although this does not affect the VM 
immediately, at some point I am guessing it syncs itself with the CMOS clock 
(which is now a large number of seconds offset from the actual time), 
resulting in a huge jump in time.  A reboot is a guaranteed way to get the 
new, incorrect time.

Although I do not understand all of the underlying code, I presume the 
correct way this should work is it should be comparing the CMOS time
that''s
just been set with the hardware clock on the physical machine, resulting in 
an offset between the two.  This would result in a generally stable number 
(ideally 0).  Obviously it is incorrect behaviour for the number to keep 
going up.  To my mind it looks like it may be somehow getting an inaccurate 
time from the system (in many cases a fixed time rather than an up-to-date 
current time).

Does anyone have any light they may be able to shed on this?  Is it possible 
it could be struggling to get an accurate time from the hardware?  I have 
checked on several occasions and both the system time and the BIOS clock are 
spot on.