William L. Thomson Jr.
2011-Feb-21 17:50 UTC
[Xen-users] domU kernel from source capable of live migration
I have been running Xen for a while, but only recently started attempting live migration. Which thus far has not worked for a variety of reasons. I am curious to know if anyone has built a domU kernel from source, using non-distro specific sources, that is capable of live migration. I have tried 2.6.36 and 2.3.37 sources without success. If anyone has compiled a kernel from either of those sources and it is capable of live migration, I would really like to see the config file. I also would not mind seeing the config file for the dom0 as well. Might be missing some stuff there. I also tried xen sources 2.6.34 for the domU kernel, but for some reason the nfs root would only come up ro and I could not get it to be rw. Hopefully the domU kernel does not have to be built using xen sources, for domU live migration. Probably could have live migrated that, but with the ro vs rw issue, system wasn''t really usable in a practical sense even if it could live migrate. Now some what unrelated I have a major problem with 2.6.37, where there are issues with PMU. Which prevents networking from working, and the system from mounting nfs root. This is not related to live migration, just preventing me from trying newer sources that might be capable of live migration. Likely a upstream issue not xen related, not sure. On 2.6.36 I get the following for PMU AMD PMU driver. ... version: 0 ... bit width: 48 ... generic registers: 4 ... value mask: 0000ffffffffffff ... max period: 00007fffffffffff ... fixed-purpose events: 0 ... event mask: 000000000000000f I use the same config from 2.6.36, but in 2.6.37 and now get Broken PMU hardware detected, software events only. Open to any feedback, comments, suggestions etc, thanks! -- William L. Thomson Jr. Obsidian-Studios, Inc. http://www.obsidian-studios.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2011-Feb-23 08:27 UTC
Re: [Xen-users] domU kernel from source capable of live migration
On Mon, Feb 21, 2011 at 12:50:26PM -0500, William L. Thomson Jr. wrote:> I have been running Xen for a while, but only recently started > attempting live migration. Which thus far has not worked for a variety > of reasons. > > I am curious to know if anyone has built a domU kernel from source, > using non-distro specific sources, that is capable of live migration. I > have tried 2.6.36 and 2.3.37 sources without success. If anyone has > compiled a kernel from either of those sources and it is capable of live > migration, I would really like to see the config file. I also would not > mind seeing the config file for the dom0 as well. Might be missing some > stuff there. >Did you try exactly 2.6.36.0 and 2.6.37.0 ? Later updates to those have many bugfixes.> I also tried xen sources 2.6.34 for the domU kernel, but for some reason > the nfs root would only come up ro and I could not get it to be rw. > Hopefully the domU kernel does not have to be built using xen sources, > for domU live migration. Probably could have live migrated that, but > with the ro vs rw issue, system wasn''t really usable in a practical > sense even if it could live migrate. >I''d recommend you to try 2.6.32.29 aswell.> Now some what unrelated I have a major problem with 2.6.37, where there > are issues with PMU. Which prevents networking from working, and the > system from mounting nfs root. This is not related to live migration, > just preventing me from trying newer sources that might be capable of > live migration. Likely a upstream issue not xen related, not sure. > > On 2.6.36 I get the following for PMU > AMD PMU driver. > ... version: 0 > ... bit width: 48 > ... generic registers: 4 > ... value mask: 0000ffffffffffff > ... max period: 00007fffffffffff > ... fixed-purpose events: 0 > ... event mask: 000000000000000f > > I use the same config from 2.6.36, but in 2.6.37 and now get > > Broken PMU hardware detected, software events only. > > Open to any feedback, comments, suggestions etc, thanks! >What kernel cmdline options are you using? You should have "console=hvc0 earlyprintk=xen" at least. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
William L. Thomson Jr.
2011-Feb-23 16:33 UTC
Re: [Xen-users] domU kernel from source capable of live migration
Thanks for the response, wasn''t sure if I should be posting this to -dev mailing list. Figured I would start with user first. On Wed, 2011-02-23 at 10:27 +0200, Pasi Kärkkäinen wrote:> > Did you try exactly 2.6.36.0 and 2.6.37.0 ? > Later updates to those have many bugfixes.Not 100% on which versions, if they were .0 or later. I am using Gentoo sources for the most part, but I have also tried regular vanilla sources via portage, which come straight from kernel.org. I am not sure the vanilla ones in portage are current, there is no 2.6.38. Let me try working with newer stuff from kernel.org directly and I will report back. I kinda have a feeling that its not possible to build a domU from source capable of live migration with regular sources at this time. Seems allot of xen support is being added into the mainstream kernel, thus likely bugs or things not 100% yet in that process.> I''d recommend you to try 2.6.32.29 aswell.Which 2.6.32.29, xen sources or kernel.org?> What kernel cmdline options are you using? > You should have "console=hvc0 earlyprintk=xen" at least.The only command line option is the console=hvc0 one, I do not have the earlyprintk=xen, and I am getting normal output. Everything is fine with the kernels up to 2.6.36, boot and run just fine. Just cannot live migrate them or even save the vm state. When I try to use a 2.6.37 kernel, it boots but lacks PMU so I get no networking, and with nfsroot, end up kernel panic unable to mount rootfs. -- William L. Thomson Jr. Obsidian-Studios, Inc. http://www.obsidian-studios.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
William L. Thomson Jr.
2011-Feb-24 23:07 UTC
Re: [Xen-users] domU kernel from source capable of live migration
On Mon, 2011-02-21 at 12:50 -0500, William L. Thomson Jr. wrote:> > Now some what unrelated I have a major problem with 2.6.37, where there > are issues with PMU. Which prevents networking from working, and the > system from mounting nfs root. This is not related to live migration, > just preventing me from trying newer sources that might be capable of > live migration. Likely a upstream issue not xen related, not sure. > > On 2.6.36 I get the following for PMU > AMD PMU driver. > ... version: 0 > ... bit width: 48 > ... generic registers: 4 > ... value mask: 0000ffffffffffff > ... max period: 00007fffffffffff > ... fixed-purpose events: 0 > ... event mask: 000000000000000f > > I use the same config from 2.6.36, but in 2.6.37 and now get > > Broken PMU hardware detected, software events only.Not sure if this is xen related or not, but I am getting the same with 2.6.38-rc6 Thinking I might need to take this up with kernel.org. Came across a post on RH''s bugzilla that is similar[1]. Doesn''t look to be using xen, not sure about other virtualization, and looks like networking might have been working for them. My missing PMU or it being detected as broken I am pretty sure is preventing my domU from being able to get networking up and running. Might be something else, not sure. Thats the only difference I can see in dmesg between a working 2.6.36 kernel and the non-working 2.6.37 and 2.6.38-rc6 kernels. Same config, but on any kernel past 2.6.36 I get no network, mounting of nfsroot fails, and no traffic is seen on the network coming from that domU in an attempt to mount nfsroot. 1. https://bugzilla.redhat.com/show_bug.cgi?id=676527 -- William L. Thomson Jr. Obsidian-Studios, Inc. http://www.obsidian-studios.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2011-Mar-23 15:44 UTC
Re: [Xen-users] domU kernel from source capable of live migration
On Thu, Feb 24, 2011 at 06:07:44PM -0500, William L. Thomson Jr. wrote:> On Mon, 2011-02-21 at 12:50 -0500, William L. Thomson Jr. wrote: > > > > Now some what unrelated I have a major problem with 2.6.37, where there > > are issues with PMU. Which prevents networking from working, and the > > system from mounting nfs root. This is not related to live migration, > > just preventing me from trying newer sources that might be capable of > > live migration. Likely a upstream issue not xen related, not sure. > > > > On 2.6.36 I get the following for PMU > > AMD PMU driver. > > ... version: 0 > > ... bit width: 48 > > ... generic registers: 4 > > ... value mask: 0000ffffffffffff > > ... max period: 00007fffffffffff > > ... fixed-purpose events: 0 > > ... event mask: 000000000000000f > > > > I use the same config from 2.6.36, but in 2.6.37 and now get > > > > Broken PMU hardware detected, software events only. > > Not sure if this is xen related or not, but I am getting the same with > 2.6.38-rc6 > > Thinking I might need to take this up with kernel.org. Came across a > post on RH''s bugzilla that is similar[1]. Doesn''t look to be using xen, > not sure about other virtualization, and looks like networking might > have been working for them. > > My missing PMU or it being detected as broken I am pretty sure is > preventing my domU from being able to get networking up and running. > Might be something else, not sure. Thats the only difference I can see > in dmesg between a working 2.6.36 kernel and the non-working 2.6.37 and > 2.6.38-rc6 kernels. > > Same config, but on any kernel past 2.6.36 I get no network, mounting of > nfsroot fails, and no traffic is seen on the network coming from that > domU in an attempt to mount nfsroot. > > 1. https://bugzilla.redhat.com/show_bug.cgi?id=676527 >Hey, Did you get this stuff working? -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
William L. Thomson Jr.
2011-Mar-26 07:57 UTC
Re: [Xen-users] domU kernel from source capable of live migration
On Wed, 2011-03-23 at 17:44 +0200, Pasi Kärkkäinen wrote:> On Thu, Feb 24, 2011 at 06:07:44PM -0500, William L. Thomson Jr. wrote: > > On Mon, 2011-02-21 at 12:50 -0500, William L. Thomson Jr. wrote:snip...> > > On 2.6.36 I get the following for PMU > > > AMD PMU driver. > > > ... version: 0 > > > ... bit width: 48 > > > ... generic registers: 4 > > > ... value mask: 0000ffffffffffff > > > ... max period: 00007fffffffffff > > > ... fixed-purpose events: 0 > > > ... event mask: 000000000000000f > > > > > > I use the same config from 2.6.36, but in 2.6.37 and now get > > > > > > Broken PMU hardware detected, software events only. > > > > Not sure if this is xen related or not, but I am getting the same with > > 2.6.38-rc6snip...> > My missing PMU or it being detected as broken I am pretty sure is > > preventing my domU from being able to get networking up and running. > > Might be something else, not sure. Thats the only difference I can see > > in dmesg between a working 2.6.36 kernel and the non-working 2.6.37 and > > 2.6.38-rc6 kernels.snip...> Hey, > > Did you get this stuff working?Well I had given up for some time, but after the inquiry I tried again yesterday with the released 2.6.38 sources. Not sure what was up with the -rc versions, but I can finally boot a kernel > 2.6.36. Its pretty odd since I just copied a config from 2.6.37 which would not boot. Ran make old config, compiled the kernel and it booted :) Now I am still getting broken PMU, and missing the stuff above, but I guess thats moot. Still not able to save or live migrate. This is what I get in xend.log when I try to save. The save file does seem to grow and be a decent size. But the vm freezes even when using xm save -c. I was also unable to restore the vm after destroying the frozen one I saved. Here is what I get in xend.log. Aside from the error with the evtchn port, looks like it did save the vm and even restored after I destroyed it. Not sure if the first failure (second line below) is whats causing the problem. Or its something relating to using a nfsroot. [2011-03-26 01:02:38 728] DEBUG (XendCheckpoint:124) [xc_save]: /usr/lib/xen/bin/xc_save 38 3 0 0 0 [2011-03-26 01:02:38 728] INFO (XendCheckpoint:423) xc_save: failed to get the suspend evtchn port [2011-03-26 01:02:38 728] INFO (XendCheckpoint:423) [2011-03-26 01:02:38 728] DEBUG (XendCheckpoint:394) suspend [2011-03-26 01:02:38 728] DEBUG (XendCheckpoint:127) In saveInputHandler suspend [2011-03-26 01:02:38 728] DEBUG (XendCheckpoint:129) Suspending 3 ... [2011-03-26 01:02:38 728] DEBUG (XendDomainInfo:519) XendDomainInfo.shutdown(suspend) [2011-03-26 01:02:38 728] DEBUG (XendDomainInfo:1891) XendDomainInfo.handleShutdownWatch [2011-03-26 01:02:38 728] DEBUG (XendDomainInfo:1891) XendDomainInfo.handleShutdownWatch [2011-03-26 01:02:38 728] INFO (XendDomainInfo:2088) Domain has shutdown: name=migrating-testvm id=3 reason=suspend. [2011-03-26 01:02:38 728] INFO (XendCheckpoint:135) Domain 3 suspended. [2011-03-26 01:02:38 728] INFO (XendCheckpoint:423) Had 0 unexplained entries in p2m table [2011-03-26 01:02:38 728] DEBUG (XendCheckpoint:144) Written done [2011-03-26 01:02:38 728] INFO (XendCheckpoint:423) Saving memory pages: iter 1 0%^H^H^H^H 5%outbuf_write: 4194304 > 4161520@12615696 [2011-03-26 01:02:38 728] INFO (XendCheckpoint:423) ^H^H^H^H 11%^H^H^H^H 17%outbuf_write: 4194304 > 4169716@12607500 [2011-03-26 01:02:38 728] INFO (XendCheckpoint:423) ^H^H^H^H 23%outbuf_write: 4194304 > 4169716@12607500 [2011-03-26 01:02:38 728] INFO (XendCheckpoint:423) ^H^H^H^H 29%^H^H^H^H 35%outbuf_write: 4194304 > 4169716@12607500 [2011-03-26 01:02:38 728] INFO (XendCheckpoint:423) ^H^H^H^H 41%outbuf_write: 4194304 > 4169716@12607500 [2011-03-26 01:02:38 728] INFO (XendCheckpoint:423) ^H^H^H^H 47%^H^H^H^H 52%outbuf_write: 4194304 > 4169716@12607500 [2011-03-26 01:02:38 728] INFO (XendCheckpoint:423) ^H^H^H^H 58%outbuf_write: 4194304 > 4169716@12607500 [2011-03-26 01:02:38 728] INFO (XendCheckpoint:423) ^H^H^H^H 64%^H^H^H^H 70%outbuf_write: 4194304 > 4169716@12607500 [2011-03-26 01:02:46 728] INFO (XendCheckpoint:423) ^H^H^H^H 76%outbuf_write: 4194304 > 4169716@12607500 [2011-03-26 01:02:46 728] INFO (XendCheckpoint:423) ^H^H^H^H 82%^H^H^H^H 88%outbuf_write: 163840 > 139252@16637964 [2011-03-26 01:02:46 728] INFO (XendCheckpoint:423) ^H^H^H^H 94%^M 1: sent 32768, skipped 0, delta 8791ms, dom0 9%, target 0%, sent 122Mb/s, dirtied 122Mb/s 32767 pages [2011-03-26 01:02:46 728] INFO (XendCheckpoint:423) Total pages sent= 32768 (0.94x) [2011-03-26 01:02:46 728] INFO (XendCheckpoint:423) (of which 0 were fixups) [2011-03-26 01:02:47 728] INFO (XendCheckpoint:423) All memory is saved [2011-03-26 01:02:50 728] INFO (XendCheckpoint:423) Save exit rc=0 [2011-03-26 01:02:50 728] DEBUG (XendDomainInfo:3117) XendDomainInfo.resumeDomain(3) [2011-03-26 01:02:50 728] DEBUG (XendDomainInfo:3158) XendDomainInfo.resumeDomain: completed [2011-03-26 01:03:15 728] DEBUG (XendDomainInfo:3053) XendDomainInfo.destroy: domid=3 -- William L. Thomson Jr. Obsidian-Studios, Inc. http://www.obsidian-studios.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users