Xen devel - May 2012 - [BUG 2.6.32.y] Broken PV migration between hosts with different uptime, non-monotonic time?

Hello,

I encountered the following bug when migrating a Linux-2.6.32.54 PV domain on 
Xen-3.4.3 between different hosts, whose uptime differs by several minutes (3 
hosts, each ~5 minutes apart): When migrating from a host with lower uptime 
to a host with higher uptime, the VM looses it''s network connection for
some
time and then continues after some minutes (roughly equivalent to the 
difference in uptime?).
There are two different symptoms: Either the VM becomes unpingable, or the VM 
is pingable but the ssh-connection freezes: a while-loop dumping /proc/uptime 
freezes and continues without a jump after the freeze is over.

When looking at the output of dmesg of the domU, I also see a jump in the 
timestamp:
[1967742.320218] eth0: no IPv6 routers present
[1968779.217256] suspending xenstore...
[1968779.217358] trying to map vcpu_info 0 at ffff88000bcbc020, mfn
85e61e, offset 32
[1968779.217358] cpu 0 using vcpu_info at ffff88000bcbc020
[ 5655.842391] suspending xenstore...
[ 5655.842477] trying to map vcpu_info 0 at ffff88000bcbc020, mfn
d5e61e, offset 32
[ 5655.842477] cpu 0 using vcpu_info at ffff88000bcbc020
[ 7745.941585] suspending xenstore...
[ 7745.941667] trying to map vcpu_info 0 at ffff88000bcbc020, mfn
be4163, offset 32
[ 7745.941667] cpu 0 using vcpu_info at ffff88000bcbc020
[342272.197261] suspending xenstore...

If I revert the following commit (original from 2.6.36-rc1), the problem does 
not show in 2.6.32.y:

commit 8a22b9996b001c88f2bfb54c6de6a05fc39e177a
Author: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Date:   Mon Jul 12 11:49:59 2010 -0700

    xen: drop xen_sched_clock in favour of using plain wallclock time
    
    xen_sched_clock only counts unstolen time.  In principle this should
    be useful to the Linux scheduler so that it knows how much time a process
    actually consumed.  But in practice this doesn''t work very well as
the
    scheduler expects the sched_clock time to be synchronized between
    cpus.  It also uses sched_clock to measure the time a task spends
    sleeping, in which case "unstolen time" isn''t meaningful.
    
    So just use plain xen_clocksource_read to return wallclock nanoseconds
    for sched_clock.

2.6.36 does not work, since 489fb49 and e7a3481 are missing: Without 
the "global synchonization point for pvclock" (AKA last_value) plus
the fix
to "reset it to 0 on resume" VMs migrate fine in the opposite
direction
(older=higher uptime → newer=lower uptime), but the original direction 
(lower→higher) now stalls for 5 minutes.

2.6.37 (which includes above patches) works fine in both directions (I only 
see a 2 second network dropout for 2 VMs going lower→higher). So something 
other must have changed also, which is missing in 2.6.32.59 so far.

I tried to unserstand all those clockevent, timer, pvclock, sched_clock() 
details, but now I''m stuck. To me it looks like xen_clocksource_read()
is not
monotonic over migration, which seems to break some assumtion of 
sched_clock() being monotonic.

Has sombody else observed a similar problem and can provide a helpful hint?
Is there anything I can look at to get this issue solved?

Sincerely
Philipp

PS: bisecting did not help much, since 2.6.32.59 contains a lot of back-ports 
from 2.6.33, 35, 36 and 37.
2.6.33 needs 281ff33 # x86_64, cpa: Don''t work hard in preserving
kernel 2M
mappings when using 4K already
2.6.33-rc1: c5cae66 fixes 65f6338  # do_suspend error handling
2.6.35-rc1: e7a3481 fixes 489fb49 # global sync point
2.6.37 needs ceff1a7 # /proc/kcore: fix seeking
-- 
Philipp Hahn           Open Source Software Engineer      hahn@univention.de
Univention GmbH        be open.                       fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen                 fax: +49 421 22 232-99
                                                   http://www.univention.de/


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On 04/05/12 09:54, Philipp Hahn wrote:
> Hello,
> 
> I encountered the following bug when migrating a Linux-2.6.32.54 PV domain
on
> Xen-3.4.3 between different hosts, whose uptime differs by several minutes
(3
> hosts, each ~5 minutes apart): When migrating from a host with lower uptime
> to a host with higher uptime, the VM looses it''s network
connection for some
> time and then continues after some minutes (roughly equivalent to the 
> difference in uptime?).
> There are two different symptoms: Either the VM becomes unpingable, or the
VM
> is pingable but the ssh-connection freezes: a while-loop dumping
/proc/uptime
> freezes and continues without a jump after the freeze is over.
Xen 3.4 doesn''t ensure that the TSC is stable across migrates (Xen 4.0
does).  If the host CPU has the CONSTANT_TSC bit in the Advanced Power
Management CPUID leaf it will pass this through to the guest which makes
the guest think the TSC is stable.

Can you try this libxc patch?

8<------------------------
libxc: clear CONSTANT_TSC bit in Advanced Power Management CPUID leaf

Even if the TSC is constant on a host, it won''t be constant across a
migrate.

Signed-off-by: David Vrabel <david.vrabel@citrix.com
----
 tools/libxc/xc_cpuid_x86.c |   14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff -r 884f24257f2c tools/libxc/xc_cpuid_x86.c
--- a/tools/libxc/xc_cpuid_x86.c	Tue Apr 17 14:47:14 2012 +0100
+++ b/tools/libxc/xc_cpuid_x86.c	Tue Apr 17 17:33:37 2012 +0100
@@ -199,6 +199,13 @@ static void xc_cpuid_hvm_policy(
             clear_bit(X86_FEATURE_NX, regs[3]);
         break;

+    case 0x80000007: /* Advanced Power Management */
+        /*
+         * Even if the TSC is constant on a host, it won''t be constant
+         * across a migrate.
+         */
+        clear_bit(X86_FEATURE_CONSTANT_TSC, regs[3]);
+        break;

     case 0x80000008:
         regs[0] &= 0x0000ffffu;
@@ -298,6 +305,13 @@ static void xc_cpuid_pv_policy(
         clear_bit(X86_FEATURE_SKINIT, regs[2]);
         clear_bit(X86_FEATURE_WDT, regs[2]);
         break;
+    case 0x80000007: /* Advanced Power Management */
+        /*
+         * Even if the TSC is constant on a host, it won''t be constant
+         * across a migrate.
+         */
+        clear_bit(X86_FEATURE_CONSTANT_TSC, regs[3]);
+        break;
     case 5: /* MONITOR/MWAIT */
     case 0xa: /* Architectural Performance Monitor Features */
     case 0x8000000a: /* SVM revision and features */

Hello David,

On Friday 04 May 2012 12:39:03 David Vrabel wrote:
> On 04/05/12 09:54, Philipp Hahn wrote:
> > I encountered the following bug when migrating a Linux-2.6.32.54 PV
> > domain on Xen-3.4.3 between different hosts, whose uptime differs by
> > several minutes
... 
> Xen 3.4 doesn''t ensure that the TSC is stable across migrates (Xen
4.0
> does).  If the host CPU has the CONSTANT_TSC bit in the Advanced Power
> Management CPUID leaf it will pass this through to the guest which makes
> the guest think the TSC is stable.
>
> Can you try this libxc patch?
>
> 8<------------------------
> libxc: clear CONSTANT_TSC bit in Advanced Power Management CPUID leaf
Excellent, that fixes the problem for me and makes perfect sense for me, since 
AFAIK 2.6.37 also received a big TSC patch which is not part of 2.6.32.y 
backport.
Thank your for your help.

Sincerely
Philipp Hahn
-- 
Philipp Hahn           Open Source Software Engineer      hahn@univention.de
Univention GmbH        be open.                       fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen                 fax: +49 421 22 232-99
                                                   http://www.univention.de/


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On 04/05/12 14:10, Philipp Hahn wrote:
> Hello David,
> 
> On Friday 04 May 2012 12:39:03 David Vrabel wrote:
>> On 04/05/12 09:54, Philipp Hahn wrote:
>>> I encountered the following bug when migrating a Linux-2.6.32.54 PV
>>> domain on Xen-3.4.3 between different hosts, whose uptime differs
by
>>> several minutes
> ... 
>> Xen 3.4 doesn''t ensure that the TSC is stable across migrates
(Xen 4.0
>> does).  If the host CPU has the CONSTANT_TSC bit in the Advanced Power
>> Management CPUID leaf it will pass this through to the guest which
makes
>> the guest think the TSC is stable.
>>
>> Can you try this libxc patch?
>>
>> 8<------------------------
>> libxc: clear CONSTANT_TSC bit in Advanced Power Management CPUID leaf
> 
> Excellent, that fixes the problem for me and makes perfect sense for me,
since
> AFAIK 2.6.37 also received a big TSC patch which is not part of 2.6.32.y 
> backport.
Thanks for testing! I could never repro the migration failures.

David