Joshua West
2008-Jan-18 02:21 UTC
[Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0
Hey all,
So it looks like the xc_save utility has issues saving a 32-bit
paravirtualized domU on a 64-bit dom0. As a result, live migration of
32-bit PV VM''s (between 64-bit dom0''s) also does not work.
However, I
am able to "xm save" and "xm migrate" 64-bit PV
domU''s on those same
64-bit dom0''s without encountering issues.
Has anybody else run into save/migrate issues of 32-bit domU''s on
64-bit
dom0''s? Do you know of a fix or workaround?
The 32-bit domU''s I''m working with have kernels which are
compiled with
PAE (HIGHMEM64G=y) support. Additionally, both of these machines are of
identical hardware configuration (Dell PowerEdge 1950''s) and have been
setup exactly the same.
Here is a snippit of output from /var/log/xen/xend.log:
#----------------------------------------------------------#
[2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:88) [xc_save]:
/usr/lib64/xen/bin/xc_save 30 1 0 0 0
[2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:333) suspend
[2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:91) In saveInputHandler
suspend
[2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:93) Suspending 1 ...
[2008-01-17 19:18:08 8353] DEBUG (XendDomainInfo:451)
XendDomainInfo.shutdown(suspend)
[2008-01-17 19:18:08 8353] DEBUG (XendDomainInfo:991)
XendDomainInfo.handleShutdownWatch
[2008-01-17 19:18:08 8353] DEBUG (XendDomainInfo:991)
XendDomainInfo.handleShutdownWatch
[2008-01-17 19:18:08 8353] INFO (XendCheckpoint:98) Domain 1 suspended.
[2008-01-17 19:18:08 8353] INFO (XendDomainInfo:1179) Domain has
shutdown: name=migrating-rhel5-test id=1 reason=suspend.
[2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:107) Written done
[2008-01-17 19:18:09 8353] INFO (XendCheckpoint:362) ERROR Internal
error: Timed out waiting for frame list updated.
[2008-01-17 19:18:09 8353] INFO (XendCheckpoint:362) ERROR Internal
error: Failed to map/save the p2m frame list
[2008-01-17 19:18:09 8353] INFO (XendCheckpoint:362) Save exit rc=1
[2008-01-17 19:18:09 8353] ERROR (XendCheckpoint:140) Save failed on
domain rhel5-test (1).
Traceback (most recent call last):
File
"/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py",
line 109, in save
forkHelper(cmd, fd, saveInputHandler, False)
File
"/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py",
line 350, in forkHelper
raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib64/xen/bin/xc_save 30 1 0 0 0 failed
[2008-01-17 19:18:09 8353] DEBUG (XendDomainInfo:1793)
XendDomainInfo.resumeDomain(1)
[2008-01-17 19:18:09 8353] DEBUG (XendCheckpoint:143)
XendCheckpoint.save: resumeDomain
#----------------------------------------------------------#
I''m not sure if this is related, but I also see messages such as the
following in "xm dmesg":
(XEN) event_channel.c:231:d3 EVTCHNOP failure: domain 1, error -17, line 231
(XEN) event_channel.c:231:d4 EVTCHNOP failure: domain 4, error -17, line 231
I suspect they''re just a result of the save failures.
Any help would be *greatly* appreciated!
Thanks!
--
Joshua West
Systems Engineer
Brandeis University
http://www.brandeis.edu
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Kurt Hackel
2008-Jan-18 06:01 UTC
Re: [Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0
Hi Joshua, Can you try the two attached patches? This should apply to 3.1.3-pre. These take care of the 32-bit dom0/64-bit domU case, but also should have some affect on your 64-bit dom0/32-bit domU. Thanks kurt On Thu, Jan 17, 2008 at 09:21:41PM -0500, Joshua West wrote:> Hey all, > > So it looks like the xc_save utility has issues saving a 32-bit > paravirtualized domU on a 64-bit dom0. As a result, live migration of > 32-bit PV VM''s (between 64-bit dom0''s) also does not work. However, I am > able to "xm save" and "xm migrate" 64-bit PV domU''s on those same 64-bit > dom0''s without encountering issues. > > Has anybody else run into save/migrate issues of 32-bit domU''s on 64-bit > dom0''s? Do you know of a fix or workaround? > > The 32-bit domU''s I''m working with have kernels which are compiled with PAE > (HIGHMEM64G=y) support. Additionally, both of these machines are of > identical hardware configuration (Dell PowerEdge 1950''s) and have been > setup exactly the same. > > Here is a snippit of output from /var/log/xen/xend.log: > > #----------------------------------------------------------# > [2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:88) [xc_save]: > /usr/lib64/xen/bin/xc_save 30 1 0 0 0 > [2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:333) suspend > [2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:91) In saveInputHandler > suspend > [2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:93) Suspending 1 ... > [2008-01-17 19:18:08 8353] DEBUG (XendDomainInfo:451) > XendDomainInfo.shutdown(suspend) > [2008-01-17 19:18:08 8353] DEBUG (XendDomainInfo:991) > XendDomainInfo.handleShutdownWatch > [2008-01-17 19:18:08 8353] DEBUG (XendDomainInfo:991) > XendDomainInfo.handleShutdownWatch > [2008-01-17 19:18:08 8353] INFO (XendCheckpoint:98) Domain 1 suspended. > [2008-01-17 19:18:08 8353] INFO (XendDomainInfo:1179) Domain has shutdown: > name=migrating-rhel5-test id=1 reason=suspend. > [2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:107) Written done > [2008-01-17 19:18:09 8353] INFO (XendCheckpoint:362) ERROR Internal error: > Timed out waiting for frame list updated. > [2008-01-17 19:18:09 8353] INFO (XendCheckpoint:362) ERROR Internal error: > Failed to map/save the p2m frame list > [2008-01-17 19:18:09 8353] INFO (XendCheckpoint:362) Save exit rc=1 > [2008-01-17 19:18:09 8353] ERROR (XendCheckpoint:140) Save failed on domain > rhel5-test (1). > Traceback (most recent call last): > File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py", line > 109, in save > forkHelper(cmd, fd, saveInputHandler, False) > File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py", line > 350, in forkHelper > raise XendError("%s failed" % string.join(cmd)) > XendError: /usr/lib64/xen/bin/xc_save 30 1 0 0 0 failed > [2008-01-17 19:18:09 8353] DEBUG (XendDomainInfo:1793) > XendDomainInfo.resumeDomain(1) > [2008-01-17 19:18:09 8353] DEBUG (XendCheckpoint:143) XendCheckpoint.save: > resumeDomain > #----------------------------------------------------------# > > I''m not sure if this is related, but I also see messages such as the > following in "xm dmesg": > (XEN) event_channel.c:231:d3 EVTCHNOP failure: domain 1, error -17, line > 231 > (XEN) event_channel.c:231:d4 EVTCHNOP failure: domain 4, error -17, line > 231 > > I suspect they''re just a result of the save failures. > > Any help would be *greatly* appreciated! > > Thanks! > > -- > Joshua West > Systems Engineer > Brandeis University > http://www.brandeis.edu > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel-- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Joshua West
2008-Jan-18 19:36 UTC
Re: [Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0
Hi Kurt, Awesome -- thanks for the patches! I''ll be testing them later on today and I''ll let you know what happens. Kurt Hackel wrote:> Hi Joshua, > > Can you try the two attached patches? This should apply to 3.1.3-pre. > These take care of the 32-bit dom0/64-bit domU case, but also should > have some affect on your 64-bit dom0/32-bit domU. > > Thanks > kurt > > > On Thu, Jan 17, 2008 at 09:21:41PM -0500, Joshua West wrote: > >> Hey all, >> >> So it looks like the xc_save utility has issues saving a 32-bit >> paravirtualized domU on a 64-bit dom0. As a result, live migration of >> 32-bit PV VM''s (between 64-bit dom0''s) also does not work. However, I am >> able to "xm save" and "xm migrate" 64-bit PV domU''s on those same 64-bit >> dom0''s without encountering issues. >> >> Has anybody else run into save/migrate issues of 32-bit domU''s on 64-bit >> dom0''s? Do you know of a fix or workaround? >> >> The 32-bit domU''s I''m working with have kernels which are compiled with PAE >> (HIGHMEM64G=y) support. Additionally, both of these machines are of >> identical hardware configuration (Dell PowerEdge 1950''s) and have been >> setup exactly the same. >> >> Here is a snippit of output from /var/log/xen/xend.log: >> >> #----------------------------------------------------------# >> [2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:88) [xc_save]: >> /usr/lib64/xen/bin/xc_save 30 1 0 0 0 >> [2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:333) suspend >> [2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:91) In saveInputHandler >> suspend >> [2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:93) Suspending 1 ... >> [2008-01-17 19:18:08 8353] DEBUG (XendDomainInfo:451) >> XendDomainInfo.shutdown(suspend) >> [2008-01-17 19:18:08 8353] DEBUG (XendDomainInfo:991) >> XendDomainInfo.handleShutdownWatch >> [2008-01-17 19:18:08 8353] DEBUG (XendDomainInfo:991) >> XendDomainInfo.handleShutdownWatch >> [2008-01-17 19:18:08 8353] INFO (XendCheckpoint:98) Domain 1 suspended. >> [2008-01-17 19:18:08 8353] INFO (XendDomainInfo:1179) Domain has shutdown: >> name=migrating-rhel5-test id=1 reason=suspend. >> [2008-01-17 19:18:08 8353] DEBUG (XendCheckpoint:107) Written done >> [2008-01-17 19:18:09 8353] INFO (XendCheckpoint:362) ERROR Internal error: >> Timed out waiting for frame list updated. >> [2008-01-17 19:18:09 8353] INFO (XendCheckpoint:362) ERROR Internal error: >> Failed to map/save the p2m frame list >> [2008-01-17 19:18:09 8353] INFO (XendCheckpoint:362) Save exit rc=1 >> [2008-01-17 19:18:09 8353] ERROR (XendCheckpoint:140) Save failed on domain >> rhel5-test (1). >> Traceback (most recent call last): >> File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py", line >> 109, in save >> forkHelper(cmd, fd, saveInputHandler, False) >> File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py", line >> 350, in forkHelper >> raise XendError("%s failed" % string.join(cmd)) >> XendError: /usr/lib64/xen/bin/xc_save 30 1 0 0 0 failed >> [2008-01-17 19:18:09 8353] DEBUG (XendDomainInfo:1793) >> XendDomainInfo.resumeDomain(1) >> [2008-01-17 19:18:09 8353] DEBUG (XendCheckpoint:143) XendCheckpoint.save: >> resumeDomain >> #----------------------------------------------------------# >> >> I''m not sure if this is related, but I also see messages such as the >> following in "xm dmesg": >> (XEN) event_channel.c:231:d3 EVTCHNOP failure: domain 1, error -17, line >> 231 >> (XEN) event_channel.c:231:d4 EVTCHNOP failure: domain 4, error -17, line >> 231 >> >> I suspect they''re just a result of the save failures. >> >> Any help would be *greatly* appreciated! >> >> Thanks! >> >> -- >> Joshua West >> Systems Engineer >> Brandeis University >> http://www.brandeis.edu >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> > > > ------------------------------------------------------------------------ > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel-- Joshua West Systems Engineer Brandeis University http://www.brandeis.edu _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Joshua West
2008-Jan-18 22:05 UTC
Re: [Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0
Hi Kurt & Xen Developers,
I applied those two patches to Xen 3.1.3-rc1. Unfortunately, the issue
still persists. xc_save fails and therefore save/migrate doesn''t work.
However, I did receive a different error message this time. Here''s the
snippit from /var/log/xen/xend.log:
#----------------------------------------------------------#
[2008-01-18 16:57:06 6774] DEBUG (XendCheckpoint:88) [xc_save]:
/usr/lib64/xen/bin/xc_save 24 1 0 0 0
[2008-01-18 16:57:06 6774] DEBUG (XendCheckpoint:333) suspend
[2008-01-18 16:57:06 6774] DEBUG (XendCheckpoint:91) In saveInputHandler
suspend
[2008-01-18 16:57:06 6774] DEBUG (XendCheckpoint:93) Suspending 1 ...
[2008-01-18 16:57:06 6774] DEBUG (XendDomainInfo:451)
XendDomainInfo.shutdown(suspend)
[2008-01-18 16:57:06 6774] DEBUG (XendDomainInfo:991)
XendDomainInfo.handleShutdownWatch
[2008-01-18 16:57:06 6774] DEBUG (XendDomainInfo:991)
XendDomainInfo.handleShutdownWatch
[2008-01-18 16:57:06 6774] INFO (XendDomainInfo:1179) Domain has
shutdown: name=migrating-rhel5-test id=1 reason=suspend.
[2008-01-18 16:57:06 6774] INFO (XendCheckpoint:98) Domain 1 suspended.
[2008-01-18 16:57:06 6774] DEBUG (XendCheckpoint:107) Written done
[2008-01-18 16:57:06 6774] INFO (XendCheckpoint:362) Had 0 unexplained
entries in p2m table
[2008-01-18 16:57:08 6774] INFO (XendCheckpoint:362) Saving memory
pages: iter 1 15%ERROR Internal error: Fatal PT race (pfn 853, type
20000000)
[2008-01-18 16:57:08 6774] INFO (XendCheckpoint:362) Save exit rc=1
[2008-01-18 16:57:08 6774] ERROR (XendCheckpoint:140) Save failed on
domain rhel5-test (1).
Traceback (most recent call last):
File
"/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py",
line 109, in save
forkHelper(cmd, fd, saveInputHandler, False)
File
"/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py",
line 350, in forkHelper
raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib64/xen/bin/xc_save 24 1 0 0 0 failed
[2008-01-18 16:57:08 6774] DEBUG (XendDomainInfo:1802)
XendDomainInfo.resumeDomain(1)
[2008-01-18 16:57:08 6774] DEBUG (XendCheckpoint:143)
XendCheckpoint.save: resumeDomain
#----------------------------------------------------------#
Just to make sure the patches aren''t the cause of this new error
message, I''m going to see what happens with Xen 3.1.3-rc1 w/o patches.
Kurt, Keir, everybody else... any ideas as to why "xm save" (and thus
"xm migrate") of 32-bit domU''s does not work on 64-bit
dom0''s? Has
anybody found this to be successful in the past?
Thanks.
Kurt Hackel wrote:> Hi Joshua,
>
> Can you try the two attached patches? This should apply to 3.1.3-pre.
> These take care of the 32-bit dom0/64-bit domU case, but also should
> have some affect on your 64-bit dom0/32-bit domU.
>
> Thanks
> kurt
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
--
Joshua West
Systems Engineer
Brandeis University
http://www.brandeis.edu
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jan-18 22:10 UTC
Re: [Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0
On 18/1/08 22:05, "Joshua West" <jwest@brandeis.edu> wrote:> Just to make sure the patches aren''t the cause of this new error > message, I''m going to see what happens with Xen 3.1.3-rc1 w/o patches. > > Kurt, Keir, everybody else... any ideas as to why "xm save" (and thus > "xm migrate") of 32-bit domU''s does not work on 64-bit dom0''s? Has > anybody found this to be successful in the past?The extent of the patches that Kurt posted indicate its pretty unlikely to work without considerable code modification. I guess those patches aren''t quite fully baked yet, unfortunately, or haven''t been written with your precise case in mind. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Joshua West
2008-Jan-18 22:13 UTC
Re: [Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0
Ahh. Well, I''m building xen-3.1.3-rc1 right now and I''ll see how that goes. Keir, have you seen this issue before? Where one cannot xm save/migrate a 32-bit domU on a 64-bit dom0? Or is this expected to work and just doesn''t go well for me? Keir Fraser wrote:> On 18/1/08 22:05, "Joshua West" <jwest@brandeis.edu> wrote: > > >> Just to make sure the patches aren''t the cause of this new error >> message, I''m going to see what happens with Xen 3.1.3-rc1 w/o patches. >> >> Kurt, Keir, everybody else... any ideas as to why "xm save" (and thus >> "xm migrate") of 32-bit domU''s does not work on 64-bit dom0''s? Has >> anybody found this to be successful in the past? >> > > The extent of the patches that Kurt posted indicate its pretty unlikely to > work without considerable code modification. I guess those patches aren''t > quite fully baked yet, unfortunately, or haven''t been written with your > precise case in mind. > > -- Keir > > >-- Joshua West Systems Engineer Brandeis University http://www.brandeis.edu _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jan-18 22:17 UTC
Re: [Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0
Looking at the issues that need to be handled to support this, it cannot be expected to work. -- Keir On 18/1/08 22:13, "Joshua West" <jwest@brandeis.edu> wrote:> Ahh. Well, I''m building xen-3.1.3-rc1 right now and I''ll see how that goes. > > Keir, have you seen this issue before? Where one cannot xm save/migrate > a 32-bit domU on a 64-bit dom0? Or is this expected to work and just > doesn''t go well for me? > > Keir Fraser wrote: >> On 18/1/08 22:05, "Joshua West" <jwest@brandeis.edu> wrote: >> >> >>> Just to make sure the patches aren''t the cause of this new error >>> message, I''m going to see what happens with Xen 3.1.3-rc1 w/o patches. >>> >>> Kurt, Keir, everybody else... any ideas as to why "xm save" (and thus >>> "xm migrate") of 32-bit domU''s does not work on 64-bit dom0''s? Has >>> anybody found this to be successful in the past? >>> >> >> The extent of the patches that Kurt posted indicate its pretty unlikely to >> work without considerable code modification. I guess those patches aren''t >> quite fully baked yet, unfortunately, or haven''t been written with your >> precise case in mind. >> >> -- Keir >> >> >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Joshua West
2008-Jan-18 23:05 UTC
Re: [Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0
Dang, doesn''t work in Xen 3.1.3-rc1 (like you said/expected), with or without Kurt''s patches. Hmm... I thought 32-bit paravirtualized domU''s on 64-bit dom0''s were supported in Xen 3.1.x? Or am I mistaken? Perhaps standard features such as start/shutdown/pause of 32-bit PV domU''s on 64-bit dom0''s are supported, but not the more advanced management features like save and migrate - and I was just unclear from the beginning? And of course... the final question that all developers hate... when s/could one _hope_ (not expect!) to see save/migrate functionality of 32-bit PV domU''s on 64-bit dom0''s? :-D Our whole reason for moving to 64-bit dom0''s was to utilize 32GB of memory, instead of the 16GB limitation on 32-bit PAE dom0''s. We can''t convert all of our virtual machines (100+) to 64-bit linux as its too much work and some actually have to be 32-bit due to vendor supplied software (Oracle, Zimbra, etc). I suppose we''ll wait to move to 64-bit until [live] migration is available, as its used quite frequently in our machine rooms. Thanks for all of your help Keir and Kurt. Keir Fraser wrote:> Looking at the issues that need to be handled to support this, it cannot be > expected to work. > > -- Keir > > On 18/1/08 22:13, "Joshua West" <jwest@brandeis.edu> wrote: > > >> Ahh. Well, I''m building xen-3.1.3-rc1 right now and I''ll see how that goes. >> >> Keir, have you seen this issue before? Where one cannot xm save/migrate >> a 32-bit domU on a 64-bit dom0? Or is this expected to work and just >> doesn''t go well for me? >> >> Keir Fraser wrote: >> >>> On 18/1/08 22:05, "Joshua West" <jwest@brandeis.edu> wrote: >>> >>> >>> >>>> Just to make sure the patches aren''t the cause of this new error >>>> message, I''m going to see what happens with Xen 3.1.3-rc1 w/o patches. >>>> >>>> Kurt, Keir, everybody else... any ideas as to why "xm save" (and thus >>>> "xm migrate") of 32-bit domU''s does not work on 64-bit dom0''s? Has >>>> anybody found this to be successful in the past? >>>> >>>> >>> The extent of the patches that Kurt posted indicate its pretty unlikely to >>> work without considerable code modification. I guess those patches aren''t >>> quite fully baked yet, unfortunately, or haven''t been written with your >>> precise case in mind. >>> >>> -- Keir >>> >>> >>> >>> > > >-- Joshua West Systems Engineer Brandeis University http://www.brandeis.edu _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jan-18 23:15 UTC
Re: [Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0
On 18/1/08 23:05, "Joshua West" <jwest@brandeis.edu> wrote:> And of course... the final question that all developers hate... when > s/could one _hope_ (not expect!) to see save/migrate functionality of > 32-bit PV domU''s on 64-bit dom0''s? :-DThe Oracle patches look like they should get a lot of the way there. Hopefully it''s just a few bugs to be shaken out of those patches. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2008-Jan-18 23:17 UTC
RE: [Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0
Joshua -- Why do you need so much memory on dom0? Are you thinking that the total memory available to guests is limited to what dom0 has? Or are you running some memory-hungry applications on dom0? Note that 64-bit Xen + 32-bit Dom0 will support both 32-bit and 64-bit domU''s. (Oracle VM uses this configuration by default on 64-bit capable machines.) And save/restore/migration has been tested to work for this (with the patch Kurt posted). Dan> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of Joshua West > Sent: Friday, January 18, 2008 4:06 PM > To: Keir Fraser > Cc: Kurt Hackel; xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] Migrate/Save of 32-bit domU Broken > on Xen 3.1.2 > 64-bit dom0 > > > Dang, doesn''t work in Xen 3.1.3-rc1 (like you said/expected), with or > without Kurt''s patches. > > Hmm... I thought 32-bit paravirtualized domU''s on 64-bit dom0''s were > supported in Xen 3.1.x? Or am I mistaken? > > Perhaps standard features such as start/shutdown/pause of 32-bit PV > domU''s on 64-bit dom0''s are supported, but not the more advanced > management features like save and migrate - and I was just > unclear from > the beginning? > > And of course... the final question that all developers hate... when > s/could one _hope_ (not expect!) to see save/migrate functionality of > 32-bit PV domU''s on 64-bit dom0''s? :-D > > Our whole reason for moving to 64-bit dom0''s was to utilize 32GB of > memory, instead of the 16GB limitation on 32-bit PAE dom0''s. We can''t > convert all of our virtual machines (100+) to 64-bit linux as its too > much work and some actually have to be 32-bit due to vendor supplied > software (Oracle, Zimbra, etc). I suppose we''ll wait to move > to 64-bit > until [live] migration is available, as its used quite > frequently in our > machine rooms. > > Thanks for all of your help Keir and Kurt. > > Keir Fraser wrote: > > Looking at the issues that need to be handled to support > this, it cannot be > > expected to work. > > > > -- Keir > > > > On 18/1/08 22:13, "Joshua West" <jwest@brandeis.edu> wrote: > > > > > >> Ahh. Well, I''m building xen-3.1.3-rc1 right now and I''ll > see how that goes. > >> > >> Keir, have you seen this issue before? Where one cannot > xm save/migrate > >> a 32-bit domU on a 64-bit dom0? Or is this expected to > work and just > >> doesn''t go well for me? > >> > >> Keir Fraser wrote: > >> > >>> On 18/1/08 22:05, "Joshua West" <jwest@brandeis.edu> wrote: > >>> > >>> > >>> > >>>> Just to make sure the patches aren''t the cause of this new error > >>>> message, I''m going to see what happens with Xen > 3.1.3-rc1 w/o patches. > >>>> > >>>> Kurt, Keir, everybody else... any ideas as to why "xm > save" (and thus > >>>> "xm migrate") of 32-bit domU''s does not work on 64-bit > dom0''s? Has > >>>> anybody found this to be successful in the past? > >>>> > >>>> > >>> The extent of the patches that Kurt posted indicate its > pretty unlikely to > >>> work without considerable code modification. I guess > those patches aren''t > >>> quite fully baked yet, unfortunately, or haven''t been > written with your > >>> precise case in mind. > >>> > >>> -- Keir > >>> > >>> > >>> > >>> > > > > > > > > > -- > Joshua West > Systems Engineer > Brandeis University > http://www.brandeis.edu > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Joshua West
2008-Jan-18 23:21 UTC
Re: [Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0
Keir Fraser wrote:> On 18/1/08 23:05, "Joshua West" <jwest@brandeis.edu> wrote: > > >> And of course... the final question that all developers hate... when >> s/could one _hope_ (not expect!) to see save/migrate functionality of >> 32-bit PV domU''s on 64-bit dom0''s? :-D >> > > The Oracle patches look like they should get a lot of the way there. > Hopefully it''s just a few bugs to be shaken out of those patches. > > -- Keir >Excellent. Kurt, I''d like to help but I''m no kernel developer. However, I can compile and test all day long like nobody''s business, so if you''re willing to push these patches through to completion, I''ll definitely help on the testing side of things. -- Joshua West Systems Engineer Brandeis University http://www.brandeis.edu _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jan-18 23:24 UTC
Re: [Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0
On 18/1/08 23:17, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:> Note that 64-bit Xen + 32-bit Dom0 will support both 32-bit > and 64-bit domU''s. (Oracle VM uses this configuration by > default on 64-bit capable machines.) And save/restore/migration > has been tested to work for this (with the patch Kurt posted).Worth checking in then? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Joshua West
2008-Jan-18 23:29 UTC
Re: [Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0
Hi Dan, Dan Magenheimer wrote:> Joshua -- > > Why do you need so much memory on dom0? Are you thinking > that the total memory available to guests is limited to > what dom0 has? Or are you running some memory-hungry > applications on dom0? >Hmm, I thought that a 32-bit dom0 system can only have up to 16GB of memory available to it and its domU''s -- and a 64-bit dom0 can have a max of 32GB. I only allocate ~512MB of memory to the dom0''s themselves. "xm info" shows us the rest of whats available for use by domU''s. Thus, even if I have 32GB of RAM in a server, the total memory that will be used in dom0 + all domU''s can only add up to 16GB. Am I just plain old wrong?> Note that 64-bit Xen + 32-bit Dom0 will support both 32-bit > and 64-bit domU''s. (Oracle VM uses this configuration by > default on 64-bit capable machines.) And save/restore/migration > has been tested to work for this (with the patch Kurt posted). >Interesting. For some reason I thought that the Hypervisor had to match the dom0''s kernel. So you''re saying that if I compile the Xen Hypervisor as 64-bit and my dom0 + domU''s both as 32-bit, I can reach the full capacity of my servers with respect to RAM (32GB) instead of being limited to 16GB? (And this only works with Kurt''s patches?).> Dan > > >> -----Original Message----- >> From: xen-devel-bounces@lists.xensource.com >> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of Joshua West >> Sent: Friday, January 18, 2008 4:06 PM >> To: Keir Fraser >> Cc: Kurt Hackel; xen-devel@lists.xensource.com >> Subject: Re: [Xen-devel] Migrate/Save of 32-bit domU Broken >> on Xen 3.1.2 >> 64-bit dom0 >> >> >> Dang, doesn''t work in Xen 3.1.3-rc1 (like you said/expected), with or >> without Kurt''s patches. >> >> Hmm... I thought 32-bit paravirtualized domU''s on 64-bit dom0''s were >> supported in Xen 3.1.x? Or am I mistaken? >> >> Perhaps standard features such as start/shutdown/pause of 32-bit PV >> domU''s on 64-bit dom0''s are supported, but not the more advanced >> management features like save and migrate - and I was just >> unclear from >> the beginning? >> >> And of course... the final question that all developers hate... when >> s/could one _hope_ (not expect!) to see save/migrate functionality of >> 32-bit PV domU''s on 64-bit dom0''s? :-D >> >> Our whole reason for moving to 64-bit dom0''s was to utilize 32GB of >> memory, instead of the 16GB limitation on 32-bit PAE dom0''s. We can''t >> convert all of our virtual machines (100+) to 64-bit linux as its too >> much work and some actually have to be 32-bit due to vendor supplied >> software (Oracle, Zimbra, etc). I suppose we''ll wait to move >> to 64-bit >> until [live] migration is available, as its used quite >> frequently in our >> machine rooms. >> >> Thanks for all of your help Keir and Kurt. >> >> Keir Fraser wrote: >> >>> Looking at the issues that need to be handled to support >>> >> this, it cannot be >> >>> expected to work. >>> >>> -- Keir >>> >>> On 18/1/08 22:13, "Joshua West" <jwest@brandeis.edu> wrote: >>> >>> >>> >>>> Ahh. Well, I''m building xen-3.1.3-rc1 right now and I''ll >>>> >> see how that goes. >> >>>> Keir, have you seen this issue before? Where one cannot >>>> >> xm save/migrate >> >>>> a 32-bit domU on a 64-bit dom0? Or is this expected to >>>> >> work and just >> >>>> doesn''t go well for me? >>>> >>>> Keir Fraser wrote: >>>> >>>> >>>>> On 18/1/08 22:05, "Joshua West" <jwest@brandeis.edu> wrote: >>>>> >>>>> >>>>> >>>>> >>>>>> Just to make sure the patches aren''t the cause of this new error >>>>>> message, I''m going to see what happens with Xen >>>>>> >> 3.1.3-rc1 w/o patches. >> >>>>>> Kurt, Keir, everybody else... any ideas as to why "xm >>>>>> >> save" (and thus >> >>>>>> "xm migrate") of 32-bit domU''s does not work on 64-bit >>>>>> >> dom0''s? Has >> >>>>>> anybody found this to be successful in the past? >>>>>> >>>>>> >>>>>> >>>>> The extent of the patches that Kurt posted indicate its >>>>> >> pretty unlikely to >> >>>>> work without considerable code modification. I guess >>>>> >> those patches aren''t >> >>>>> quite fully baked yet, unfortunately, or haven''t been >>>>> >> written with your >> >>>>> precise case in mind. >>>>> >>>>> -- Keir >>>>> >>>>> >>>>> >>>>> >>>>> >>> >>> >> -- >> Joshua West >> Systems Engineer >> Brandeis University >> http://www.brandeis.edu >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> >> > >-- Joshua West Systems Engineer Brandeis University http://www.brandeis.edu _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Joshua West
2008-Jan-18 23:32 UTC
Re: [Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0
Joshua West wrote:> Hi Dan, > > Dan Magenheimer wrote: >> Joshua -- >> >> Why do you need so much memory on dom0? Are you thinking >> that the total memory available to guests is limited to >> what dom0 has? Or are you running some memory-hungry >> applications on dom0? >> > > Hmm, I thought that a 32-bit dom0 system can only have up to 16GB of > memory available to it and its domU''s -- and a 64-bit dom0 can have a > max of 32GB. I only allocate ~512MB of memory to the dom0''s > themselves. "xm info" shows us the rest of whats available for use by > domU''s. > > Thus, even if I have 32GB of RAM in a server, the total memory that > will be used in dom0 + all domU''s can only add up to 16GB. Am I just > plain old wrong? > >> Note that 64-bit Xen + 32-bit Dom0 will support both 32-bit >> and 64-bit domU''s. (Oracle VM uses this configuration by >> default on 64-bit capable machines.) And save/restore/migration >> has been tested to work for this (with the patch Kurt posted). >> > > Interesting. For some reason I thought that the Hypervisor had to > match the dom0''s kernel. So you''re saying that if I compile the Xen > Hypervisor as 64-bit and my dom0 + domU''s both as 32-bit, I can reach > the full capacity of my servers with respect to RAM (32GB) instead of > being limited to 16GB? (And this only works with Kurt''s patches?).Sorry about this second question. Looks like you already answered it in your first message :-/ I''m going to give it a whirl (64-bit Xen Hypervisor, 32-bit dom0, 64-bit and 32-bit domU''s -- all with Kurt''s patches). Thanks.> >> Dan > >-- Joshua West Systems Engineer Brandeis University http://www.brandeis.edu _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Kurt Hackel
2008-Jan-19 00:12 UTC
Re: [Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0
On Fri, Jan 18, 2008 at 11:24:05PM +0000, Keir Fraser wrote:> On 18/1/08 23:17, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote: > > > Note that 64-bit Xen + 32-bit Dom0 will support both 32-bit > > and 64-bit domU''s. (Oracle VM uses this configuration by > > default on 64-bit capable machines.) And save/restore/migration > > has been tested to work for this (with the patch Kurt posted). > > Worth checking in then? > > -- KeirIf you''re willing to take the patches as-is to support 32-bit dom0/64-bit guest case, yes. We''ll have to continue working on the opposite case, but at least there is some way of supporting save/restore/migrate for all guest types today. thanks kurt -- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jan-19 08:04 UTC
Re: [Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0
On 18/1/08 23:29, "Joshua West" <jwest@brandeis.edu> wrote:> Thus, even if I have 32GB of RAM in a server, the total memory that will > be used in dom0 + all domU''s can only add up to 16GB. Am I just plain > old wrong? > >> Note that 64-bit Xen + 32-bit Dom0 will support both 32-bit >> and 64-bit domU''s. (Oracle VM uses this configuration by >> default on 64-bit capable machines.) And save/restore/migration >> has been tested to work for this (with the patch Kurt posted). > > Interesting. For some reason I thought that the Hypervisor had to match > the dom0''s kernel. So you''re saying that if I compile the Xen > Hypervisor as 64-bit and my dom0 + domU''s both as 32-bit, I can reach > the full capacity of my servers with respect to RAM (32GB) instead of > being limited to 16GB? (And this only works with Kurt''s patches?).As long as you run 64-bit Xen you can run 32-bit dom0 and domUs and use the full memory of your system. The 32-bit guests are not even limited to 16GB each, let alone jointly. This is true even without Kurt''s patches. What Kurt''s patches fix is save/restore/migrate of a 64-bit domU from a 32-bit dom0. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Joshua West
2008-Jan-20 21:28 UTC
Re: [Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0
Kurt Hackel wrote:> On Fri, Jan 18, 2008 at 11:24:05PM +0000, Keir Fraser wrote: > >> On 18/1/08 23:17, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote: >> >> >>> Note that 64-bit Xen + 32-bit Dom0 will support both 32-bit >>> and 64-bit domU''s. (Oracle VM uses this configuration by >>> default on 64-bit capable machines.) And save/restore/migration >>> has been tested to work for this (with the patch Kurt posted). >>> >> Worth checking in then? >> >> -- Keir >> > > If you''re willing to take the patches as-is to support 32-bit > dom0/64-bit guest case, yes. We''ll have to continue working on the > opposite case, but at least there is some way of supporting > save/restore/migrate for all guest types today. > > thanks > kurt >Hey Keir, Any thoughts on if these patches will be accepted for 3.1.3? -- Joshua West Systems Engineer Brandeis University http://www.brandeis.edu _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jan-21 07:59 UTC
Re: [Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.2 64-bit dom0
On 20/1/08 21:28, "Joshua West" <jwest@brandeis.edu> wrote:>> If you''re willing to take the patches as-is to support 32-bit >> dom0/64-bit guest case, yes. We''ll have to continue working on the >> opposite case, but at least there is some way of supporting >> save/restore/migrate for all guest types today. >> >> thanks >> kurt >> > > Hey Keir, > > Any thoughts on if these patches will be accepted for 3.1.3?Unlikely. 3.1 branch is now two steps behind development tip. It''s in the freezer. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Bruce Rogers
2008-Feb-04 21:08 UTC
Re: [Xen-devel] Migrate/Save of 32-bit domU Broken on Xen 3.1.264-bit dom0
>>> On 1/18/2008 at 5:12 PM, Kurt Hackel <kurt.hackel@oracle.com> wrote: > On Fri, Jan 18, 2008 at 11:24:05PM +0000, Keir Fraser wrote: >> On 18/1/08 23:17, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote: >> >> > Note that 64-bit Xen + 32-bit Dom0 will support both 32-bit >> > and 64-bit domU''s. (Oracle VM uses this configuration by >> > default on 64-bit capable machines.) And save/restore/migration >> > has been tested to work for this (with the patch Kurt posted). >> >> Worth checking in then? >> >> -- Keir > > If you''re willing to take the patches as-is to support 32-bit > dom0/64-bit guest case, yes. We''ll have to continue working on the > opposite case, but at least there is some way of supporting > save/restore/migrate for all guest types today. > > thanks > kurtHas there been any further progress made on either case? - Bruce _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel