Pim van Riezen
2011-Feb-04 11:26 UTC
[Xen-devel] Wonky results with pvops live migration under 4.0.1
Good day, We migrated one of our Xen clusters to the gitco 4.0.1 release after seeing that it seemed to fix the live migration issues we were having under 3.4.3 on our test cluster. It turns out our tests had no bearing on our production cluster. The production cluster is a mix of mostly AMD Opteron with a few Intel Xeon. Migration issues do not correlate to cpu architectures, however. I have run a test with a machine running pvops 2.6.32.25 performing live migrations. For some source & destination pairs the migration would consistently work. For other combinations it would consistently break. The symptoms when it breaks are that the console does not respond to keyboard input. The network does ping. Interactive ssh sessions no longer work. The shell is loaded, but it also does not respond to keyboard input. Noninteractive ssh calls are normally executed. If a shutdown is sent to the vps, the shutdown sequence seems to hang on running sync. All dom0 nodes run CentOS 5.5 with gitco Xen 4.0.1 repositories. Here is a raw dump of some migration tests: A = 16 cores Xeon, 48GB memory, Emulex FC 8Gb B-E = 24 cores Opteron, 128GB memory, Emulex FC 8Gb C->B OK B->C FAIL B->C FAIL C->B OK D->C OK C->D FAIL C->D FAIL B->D FAIL D->B OK E->F OK F->E FAIL F->E FAIL E->A OK A->E FAIL The guest has 256MB memory, all dom0s are configured in bridging mode (with the bridges attached to a vlan interface). Is there anything I should try? Cheers, Pim van Riezen _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pim van Riezen
2011-Feb-04 11:28 UTC
Re: [Xen-devel] Wonky results with pvops live migration under 4.0.1
Forgot to mention, in both ok and failed cases, xend logs a succesful snapshot/restore on both sides of the migration. On Feb 4, 2011, at 12:26 , Pim van Riezen wrote:> Good day, > > We migrated one of our Xen clusters to the gitco 4.0.1 release after seeing that it seemed to fix the live migration issues we were having under 3.4.3 on our test cluster. It turns out our tests had no bearing on our production cluster. > > The production cluster is a mix of mostly AMD Opteron with a few Intel Xeon. Migration issues do not correlate to cpu architectures, however. > > I have run a test with a machine running pvops 2.6.32.25 performing live migrations. For some source & destination pairs the migration would consistently work. For other combinations it would consistently break. > > The symptoms when it breaks are that the console does not respond to keyboard input. The network does ping. Interactive ssh sessions no longer work. The shell is loaded, but it also does not respond to keyboard input. Noninteractive ssh calls are normally executed. If a shutdown is sent to the vps, the shutdown sequence seems to hang on running sync. > > All dom0 nodes run CentOS 5.5 with gitco Xen 4.0.1 repositories. Here is a raw dump of some migration tests: > > A = 16 cores Xeon, 48GB memory, Emulex FC 8Gb > B-E = 24 cores Opteron, 128GB memory, Emulex FC 8Gb > > C->B OK > B->C FAIL > B->C FAIL > C->B OK > D->C OK > C->D FAIL > C->D FAIL > B->D FAIL > D->B OK > E->F OK > F->E FAIL > F->E FAIL > E->A OK > A->E FAIL > > The guest has 256MB memory, all dom0s are configured in bridging mode (with the bridges attached to a vlan interface). > > Is there anything I should try? > > Cheers, > Pim van Riezen > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2011-Feb-04 11:40 UTC
Re: [Xen-devel] Wonky results with pvops live migration under 4.0.1
On Fri, 2011-02-04 at 11:26 +0000, Pim van Riezen wrote:> > I have run a test with a machine running pvops 2.6.32.25 performing > live migrations. For some source & destination pairs the migration > would consistently work. For other combinations it would consistently > break.Please check the xen-devel archives for the thread "Live migration bug introduced in 2.6.32.16?" from 27/1/2011. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel