thr3ads.net - Xen devel - [Xen-devel] Clock jumping during live migrations [Oct 2011]

If this information is useful, please help other people find it:
Share via:

Nathan March

2011-Oct-26 22:31 UTC

[Xen-devel] Clock jumping during live migrations

Hi everyone,

Having an issue where the clock on a VM will jump ahead by the timezone 
offset but only certain directions between certain machines.

For example, if I run a "sleep 1; date" loop on the VM and migrate it 
from xen4 to xen7:

Wed Oct 26 21:14:46 PDT 2011
Thu Oct 27 04:19:03 PDT 2011

Trying multiple times with the same dom0''s it seems to be consistently
7
hours (PST offset maybe?) 4 minutes  and 17 seconds:

Thu Oct 27 04:23:27 PDT 2011
Thu Oct 27 11:27:45 PDT 2011

Thu Oct 27 11:43:01 PDT 2011
Thu Oct 27 18:47:18 PDT 2011

If I keep migrating back and forth it keeps going ahead each time, it 
never snaps back. Those were both migrating from xen4 -> xen7, migrating 
the VM back to xen4 does not result in a clock change of any noticeable 
amount.

I''ve tried using hwclock to make sure the RTC''s are in sync
with the
system clock but it didn''t seem to make a difference (although they
were
wrong initially, unless xen requires a reboot to pick up a change here).

If I try from xen8 to xen7 it does it, but by a much smaller amount... 
again, going the reverse direction causes no change.

Thu Oct 27 18:50:08 PDT 2011
Thu Oct 27 18:51:57 PDT 2011

and again the time between multiple runs seems to be a consistent 1:50:

Fri Oct 28 16:16:27 PDT 2011
Fri Oct 28 16:18:17 PDT 2011

All the dom0''s are identical as far as I''m aware except that
some have
hyperthreading on and some have it off. I''ve seen this issue between
two
hyperthreading enabled hosts so doesn''t seem to be specific to moving 
between them.

I''ve tested this with the VM using both the xen and the tsc clocksource
but it made no difference.

Other info for xen4 / xen7:

xen4 ~ # uname -a
Linux xen4 3.0.3 #3 SMP Thu Sep 1 23:39:43 PDT 2011 x86_64 Intel(R) 
Xeon(R) CPU L5640 @ 2.27GHz GenuineIntel GNU/Linux
xen4 ~ # uptime
  14:48:33 up 1 day,  6:03,  2 users,  load average: 0.00, 0.01, 0.05
xen4 ~ # xm info
host                   : xen4
release                : 3.0.3
version                : #3 SMP Thu Sep 1 23:39:43 PDT 2011
machine                : x86_64
nr_cpus                : 12
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 1
cpu_mhz                : 2266
hw_caps                : 
bfebfbff:2c100800:00000000:00003f40:009ee3fd:00000000:00000001:00000000
virt_caps              : hvm hvm_directio
total_memory           : 98294
free_memory            : 87822
free_cpus              : 0
xen_major              : 4
xen_minor              : 1
xen_extra              : .1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
xen_commandline        : console=com1,com2,vga com1=115200,8n1 
com2=115200,8n1 dom0_mem=1024M dom0_max_vcpus=1 dom0_vcpus_pin=true
cc_compiler            : gcc version 4.3.4 (Gentoo 4.3.4 p1.1, pie-10.1.5)
cc_compile_by          : root
cc_compile_domain      : nmsrv.com
cc_compile_date        : Thu Sep  1 23:34:06 PDT 2011
xend_config_format     : 4
xen4 ~ # cat /etc/timezone
PST8PDT
xen4 ~ # md5sum /etc/localtime
c9452f6b9e08d83c6815c38600798964  /etc/localtime

xen7 ~ # uname -a
Linux xen7 3.0.3 #2 SMP Tue Aug 23 12:58:39 PDT 2011 x86_64 Intel(R) 
Xeon(R) CPU L5640 @ 2.27GHz GenuineIntel GNU/Linux
xen7 ~ # uptime
  14:48:34 up 42 days, 23:38,  3 users,  load average: 0.04, 0.08, 0.13
xen7 ~ # xm info
host                   : xen7
release                : 3.0.3
version                : #2 SMP Tue Aug 23 12:58:39 PDT 2011
machine                : x86_64
nr_cpus                : 24
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 2
cpu_mhz                : 2266
hw_caps                : 
bfebfbff:2c100800:00000000:00003f40:009ee3fd:00000000:00000001:00000000
virt_caps              : hvm hvm_directio
total_memory           : 98294
free_memory            : 24329
free_cpus              : 0
xen_major              : 4
xen_minor              : 1
xen_extra              : .1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
xen_commandline        : console=com1,com2,vga com1=115200,8n1 
com2=115200,8n1 dom0_mem=1024M dom0_max_vcpus=1 dom0_vcpus_pin=true
cc_compiler            : gcc version 4.3.4 (Gentoo 4.3.4 p1.1, pie-10.1.5)
cc_compile_by          : root
cc_compile_domain      : nmsrv.com
cc_compile_date        : Mon Aug 22 11:28:50 PDT 2011
xend_config_format     : 4
xen7 ~ # cat /etc/timezone
PST8PDT
xen7 ~ # md5sum /etc/localtime
c9452f6b9e08d83c6815c38600798964  /etc/localtime

and the test VM i''ve been using:

nathanxen1 ~ # uname -a
Linux nathanxen1 2.6.32.27-grsec-gt-r2 #1 SMP Wed Aug 10 17:13:56 PDT 
2011 x86_64 Intel(R) Xeon(R) CPU L5640 @ 2.27GHz GenuineIntel GNU/Linux
nathanxen1 ~ # cat 
/sys/devices/system/clocksource/clocksource0/available_clocksource
xen tsc
nathanxen1 ~ # cat 
/sys/devices/system/clocksource/clocksource0/current_clocksource
tsc

Any suggestions?

Unfortunately I don''t know when this bug appeared, it used to work fine
so not sure what I upgraded to give me an idea of when it was introduced.

Thanks,
Nathan

-- 
Nathan March<nathan@gt.net>
Gossamer Threads Inc. http://www.gossamer-threads.com/
Tel: (604) 687-5804 Fax: (604) 687-5806


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Tim Deegan

2011-Oct-27 11:07 UTC

head link

Re: [Xen-devel] Clock jumping during live migrations

Hi, 

At 15:31 -0700 on 26 Oct (1319643098), Nathan March
wrote:> Having an issue where the clock on a VM will jump ahead by the timezone 
> offset but only certain directions between certain machines.
> 
> For example, if I run a "sleep 1; date" loop on the VM and
migrate it
> from xen4 to xen7:
> 
> Wed Oct 26 21:14:46 PDT 2011
> Thu Oct 27 04:19:03 PDT 2011
...
> xen4 ~ # uname -a
> Linux xen4 3.0.3 #3 SMP Thu Sep 1 23:39:43 PDT 2011 x86_64 Intel(R) 
I think maybe you need these patches on your dom0 kernel:

http://lists.xensource.com/archives/html/xen-devel/2011-10/msg01762.html

Tim.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Nathan March

2012-Jun-18 21:22 UTC

head link

Re: Clock jumping during live migrations

On 10/27/2011 4:07 AM, Tim Deegan wrote:> At 15:31 -0700 on 26 Oct (1319643098), Nathan March wrote:
>> Having an issue where the clock on a VM will jump ahead by the timezone
>> offset but only certain directions between certain machines.
>>
>> For example, if I run a "sleep 1; date" loop on the VM and
migrate it
>> from xen4 to xen7:
>>
>> Wed Oct 26 21:14:46 PDT 2011
>> Thu Oct 27 04:19:03 PDT 2011
>> xen4 ~ # uname -a
>> Linux xen4 3.0.3 #3 SMP Thu Sep 1 23:39:43 PDT 2011 x86_64 Intel(R)
> I think maybe you need these patches on your dom0 kernel:
>
> http://lists.xensource.com/archives/html/xen-devel/2011-10/msg01762.html
Unfortunately I''m still running into this problem, migrating
VM''s
between particular dom0''s will result in a jump forward of anywhere
from
minutes to hours. This has persisted through dom0 and xen upgrades (now 
on dom0 3.2.7 / xen 4.1.2) so maybe we''re missing something in the domu
kernel (2.6.32.27)? Reviewing kernel change logs doesn''t indicate 
anything''s changed in regard to clocksources though.

Examples (just running date on the domU after each migration)

Initially: Mon Jun 18 13:53:48 PDT 2012
Xen2 -> Xen8: Mon Jun 18 13:55:05 PDT 2012 (No shift)
Xen8 -> Xen2: Mon Jun 18 14:27:31 PDT 2012 (+30 approx minutes)
Xen2 -> Xen8: Mon Jun 18 14:28:38 PDT 2012 (No shift)
Xen8 -> Xen2: Mon Jun 18 15:01:21 PDT 2012 (+30 minutes again)

This is now running with a domu kernel of 2.6.32.27 (w/ grsec) and a 
dom0 kernel of 3.2.7 on both dom0''s (they should be identical).
I''ve
tried both the "tsc" and "xen" clocksources in the domU with
no effect
(no others are listed in available_clocksources).

Not sure if this is useful, but while on xen8:
(XEN) dom27: mode=0,ofs=0xffc56b8eee259ecf,khz=2266790,inc=13,vtsc 
count: 37835 kernel, 750 user

After moving to xen2:
(XEN) dom7: mode=0,ofs=0xffc51d4984ee8f42,khz=2266790,inc=14,vtsc count: 
17628 kernel, 71 user

Anyone have any suggestions here?

Thanks,
Nathan

-- 
Nathan March<nathan@gt.net>
Gossamer Threads Inc. http://www.gossamer-threads.com/
Tel: (604) 687-5804 Fax: (604) 687-5806

Xen devel - Oct 2011 - Clock jumping during live migrations

[Xen-devel] Clock jumping during live migrations

Re: [Xen-devel] Clock jumping during live migrations

Re: Clock jumping during live migrations