Philipp Hahn
2012-May-04 08:54 UTC
[BUG 2.6.32.y] Broken PV migration between hosts with different uptime, non-monotonic time?
Hello, I encountered the following bug when migrating a Linux-2.6.32.54 PV domain on Xen-3.4.3 between different hosts, whose uptime differs by several minutes (3 hosts, each ~5 minutes apart): When migrating from a host with lower uptime to a host with higher uptime, the VM looses it''s network connection for some time and then continues after some minutes (roughly equivalent to the difference in uptime?). There are two different symptoms: Either the VM becomes unpingable, or the VM is pingable but the ssh-connection freezes: a while-loop dumping /proc/uptime freezes and continues without a jump after the freeze is over. When looking at the output of dmesg of the domU, I also see a jump in the timestamp: [1967742.320218] eth0: no IPv6 routers present [1968779.217256] suspending xenstore... [1968779.217358] trying to map vcpu_info 0 at ffff88000bcbc020, mfn 85e61e, offset 32 [1968779.217358] cpu 0 using vcpu_info at ffff88000bcbc020 [ 5655.842391] suspending xenstore... [ 5655.842477] trying to map vcpu_info 0 at ffff88000bcbc020, mfn d5e61e, offset 32 [ 5655.842477] cpu 0 using vcpu_info at ffff88000bcbc020 [ 7745.941585] suspending xenstore... [ 7745.941667] trying to map vcpu_info 0 at ffff88000bcbc020, mfn be4163, offset 32 [ 7745.941667] cpu 0 using vcpu_info at ffff88000bcbc020 [342272.197261] suspending xenstore... If I revert the following commit (original from 2.6.36-rc1), the problem does not show in 2.6.32.y: commit 8a22b9996b001c88f2bfb54c6de6a05fc39e177a Author: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com> Date: Mon Jul 12 11:49:59 2010 -0700 xen: drop xen_sched_clock in favour of using plain wallclock time xen_sched_clock only counts unstolen time. In principle this should be useful to the Linux scheduler so that it knows how much time a process actually consumed. But in practice this doesn''t work very well as the scheduler expects the sched_clock time to be synchronized between cpus. It also uses sched_clock to measure the time a task spends sleeping, in which case "unstolen time" isn''t meaningful. So just use plain xen_clocksource_read to return wallclock nanoseconds for sched_clock. 2.6.36 does not work, since 489fb49 and e7a3481 are missing: Without the "global synchonization point for pvclock" (AKA last_value) plus the fix to "reset it to 0 on resume" VMs migrate fine in the opposite direction (older=higher uptime → newer=lower uptime), but the original direction (lower→higher) now stalls for 5 minutes. 2.6.37 (which includes above patches) works fine in both directions (I only see a 2 second network dropout for 2 VMs going lower→higher). So something other must have changed also, which is missing in 2.6.32.59 so far. I tried to unserstand all those clockevent, timer, pvclock, sched_clock() details, but now I''m stuck. To me it looks like xen_clocksource_read() is not monotonic over migration, which seems to break some assumtion of sched_clock() being monotonic. Has sombody else observed a similar problem and can provide a helpful hint? Is there anything I can look at to get this issue solved? Sincerely Philipp PS: bisecting did not help much, since 2.6.32.59 contains a lot of back-ports from 2.6.33, 35, 36 and 37. 2.6.33 needs 281ff33 # x86_64, cpa: Don''t work hard in preserving kernel 2M mappings when using 4K already 2.6.33-rc1: c5cae66 fixes 65f6338 # do_suspend error handling 2.6.35-rc1: e7a3481 fixes 489fb49 # global sync point 2.6.37 needs ceff1a7 # /proc/kcore: fix seeking -- Philipp Hahn Open Source Software Engineer hahn@univention.de Univention GmbH be open. fon: +49 421 22 232- 0 Mary-Somerville-Str.1 D-28359 Bremen fax: +49 421 22 232-99 http://www.univention.de/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
David Vrabel
2012-May-04 10:39 UTC
Re: [BUG 2.6.32.y] Broken PV migration between hosts with different uptime, non-monotonic time?
On 04/05/12 09:54, Philipp Hahn wrote:> Hello, > > I encountered the following bug when migrating a Linux-2.6.32.54 PV domain on > Xen-3.4.3 between different hosts, whose uptime differs by several minutes (3 > hosts, each ~5 minutes apart): When migrating from a host with lower uptime > to a host with higher uptime, the VM looses it''s network connection for some > time and then continues after some minutes (roughly equivalent to the > difference in uptime?). > There are two different symptoms: Either the VM becomes unpingable, or the VM > is pingable but the ssh-connection freezes: a while-loop dumping /proc/uptime > freezes and continues without a jump after the freeze is over.Xen 3.4 doesn''t ensure that the TSC is stable across migrates (Xen 4.0 does). If the host CPU has the CONSTANT_TSC bit in the Advanced Power Management CPUID leaf it will pass this through to the guest which makes the guest think the TSC is stable. Can you try this libxc patch? 8<------------------------ libxc: clear CONSTANT_TSC bit in Advanced Power Management CPUID leaf Even if the TSC is constant on a host, it won''t be constant across a migrate. Signed-off-by: David Vrabel <david.vrabel@citrix.com ---- tools/libxc/xc_cpuid_x86.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff -r 884f24257f2c tools/libxc/xc_cpuid_x86.c --- a/tools/libxc/xc_cpuid_x86.c Tue Apr 17 14:47:14 2012 +0100 +++ b/tools/libxc/xc_cpuid_x86.c Tue Apr 17 17:33:37 2012 +0100 @@ -199,6 +199,13 @@ static void xc_cpuid_hvm_policy( clear_bit(X86_FEATURE_NX, regs[3]); break; + case 0x80000007: /* Advanced Power Management */ + /* + * Even if the TSC is constant on a host, it won''t be constant + * across a migrate. + */ + clear_bit(X86_FEATURE_CONSTANT_TSC, regs[3]); + break; case 0x80000008: regs[0] &= 0x0000ffffu; @@ -298,6 +305,13 @@ static void xc_cpuid_pv_policy( clear_bit(X86_FEATURE_SKINIT, regs[2]); clear_bit(X86_FEATURE_WDT, regs[2]); break; + case 0x80000007: /* Advanced Power Management */ + /* + * Even if the TSC is constant on a host, it won''t be constant + * across a migrate. + */ + clear_bit(X86_FEATURE_CONSTANT_TSC, regs[3]); + break; case 5: /* MONITOR/MWAIT */ case 0xa: /* Architectural Performance Monitor Features */ case 0x8000000a: /* SVM revision and features */
Philipp Hahn
2012-May-04 13:10 UTC
Re: [BUG 2.6.32.y] Broken PV migration between hosts with different uptime, non-monotonic time?
Hello David, On Friday 04 May 2012 12:39:03 David Vrabel wrote:> On 04/05/12 09:54, Philipp Hahn wrote: > > I encountered the following bug when migrating a Linux-2.6.32.54 PV > > domain on Xen-3.4.3 between different hosts, whose uptime differs by > > several minutes...> Xen 3.4 doesn''t ensure that the TSC is stable across migrates (Xen 4.0 > does). If the host CPU has the CONSTANT_TSC bit in the Advanced Power > Management CPUID leaf it will pass this through to the guest which makes > the guest think the TSC is stable. > > Can you try this libxc patch? > > 8<------------------------ > libxc: clear CONSTANT_TSC bit in Advanced Power Management CPUID leafExcellent, that fixes the problem for me and makes perfect sense for me, since AFAIK 2.6.37 also received a big TSC patch which is not part of 2.6.32.y backport. Thank your for your help. Sincerely Philipp Hahn -- Philipp Hahn Open Source Software Engineer hahn@univention.de Univention GmbH be open. fon: +49 421 22 232- 0 Mary-Somerville-Str.1 D-28359 Bremen fax: +49 421 22 232-99 http://www.univention.de/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
David Vrabel
2012-May-04 13:20 UTC
Re: [BUG 2.6.32.y] Broken PV migration between hosts with different uptime, non-monotonic time?
On 04/05/12 14:10, Philipp Hahn wrote:> Hello David, > > On Friday 04 May 2012 12:39:03 David Vrabel wrote: >> On 04/05/12 09:54, Philipp Hahn wrote: >>> I encountered the following bug when migrating a Linux-2.6.32.54 PV >>> domain on Xen-3.4.3 between different hosts, whose uptime differs by >>> several minutes > ... >> Xen 3.4 doesn''t ensure that the TSC is stable across migrates (Xen 4.0 >> does). If the host CPU has the CONSTANT_TSC bit in the Advanced Power >> Management CPUID leaf it will pass this through to the guest which makes >> the guest think the TSC is stable. >> >> Can you try this libxc patch? >> >> 8<------------------------ >> libxc: clear CONSTANT_TSC bit in Advanced Power Management CPUID leaf > > Excellent, that fixes the problem for me and makes perfect sense for me, since > AFAIK 2.6.37 also received a big TSC patch which is not part of 2.6.32.y > backport.Thanks for testing! I could never repro the migration failures. David
Maybe Matching Threads
- when timer go back in dom0 save and restore or migrate, PV domain hung
- Re: RE: Re: Re: when timer go back in dom0 save and restore ormigrate, PV domain hung
- [PATCH,RFC 6/17] 32-on-64 shared info handling
- [PATCH 4/6] X86: Disable PCID/INVPCID for pv
- [PATCH 3/3] Expose tsc adjust to hvm guest