I am seeing a problem when live migrating a guest under unstable when I
have xm configured to XenAPI.
If my xm-config.xml file has the XenAPI stuff commented out - live
migrate (xm migrate --live <domid> <dest IP>) seems to be working in
my
test environment.
However, if I enable it - I get failures when migrating the same guest
(see dump from xend.log below.) They obviously go down quite different
code paths in these two cases.
Being somewhat new to the xm, and xenapi control stacks - I was hoping
someone might point me in the right direction to track down what might
be happening here...
Any help is appreciated.
Ben
Xend log snippet:
2007-11-07 09:01:35 15422] DEBUG (XendCheckpoint:88) [xc_save]:
/usr/lib64/xen/bin/xc_save 20 2 0 0 5
[2007-11-07 09:01:35 15422] INFO (XendCheckpoint:362) Saving memory
pages: iter 1 0%ERROR Internal error: Error when writing to state file
(5) (errno 104)
[2007-11-07 09:01:35 15422] INFO (XendCheckpoint:362) Save exit rc=1
[2007-11-07 09:01:36 15422] ERROR (XendCheckpoint:140) Save failed on
domain VS022 (2).
Traceback (most recent call last):
File "/usr/lib64/python/xen/xend/XendCheckpoint.py", line 108, in
save
forkHelper(cmd, fd, saveInputHandler, False)
File "/usr/lib64/python/xen/xend/XendCheckpoint.py", line 350, in
forkHelper
raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib64/xen/bin/xc_save 20 2 0 0 5 failed
[2007-11-07 09:01:36 15422] DEBUG (XendDomainInfo:1856)
XendDomainInfo.resumeDomain(2)
[2007-11-07 09:01:36 15422] ERROR (xmlrpclib2:166) Internal error
handling VM.migrate
Traceback (most recent call last):
File "/usr/lib64/python/xen/util/xmlrpclib2.py", line 131, in
_marshaled_dispatch
response = self._dispatch(method, params)
File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line 406, in
_dispatch
return func(*params)
File "/usr/lib64/python/xen/xend/XendAPI.py", line 221, in f
return func(self, *args, **kwargs)
File "/usr/lib64/python/xen/xend/XendAPI.py", line 256, in
check_session
return func(self, session, *args, **kwargs)
File "/usr/lib64/python/xen/xend/XendAPI.py", line 312, in
<lambda>
''VM'', func, *args, **kwargs)
File "/usr/lib64/python/xen/xend/XendAPI.py", line 268, in _check_ref
return func(api, session, ref, *args, **kwargs)
File "/usr/lib64/python/xen/xend/XendAPI.py", line 1763, in
VM_migrate
bool(live), resource, port)
File "/usr/lib64/python/xen/xend/XendDomain.py", line 1282, in
domain_migrate
XendCheckpoint.save(sock.fileno(), dominfo, True, live, dst)
File "/usr/lib64/python/xen/xend/XendCheckpoint.py", line 142, in
save
dominfo.resumeDomain()
File "/usr/lib64/python/xen/xend/XendDomainInfo.py", line 1885, in
resumeDomain
if self.is_hvm():
File "/usr/lib64/python/xen/xend/XendDomainInfo.py", line 2139, in
__getattr__
raise AttributeError()
AttributeError
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Ben Guthro
2007-Nov-08 14:32 UTC
[Xen-devel] Re: Live Migration problems using xenapi with xm
I''ve managed to track this down to being a bug in HVM migration using XenAPI. It seems that restore is trying to restore a PV guest, while save is trying to save an HVM guest... so the protocol of how many mytes to read on the socket is off, and naturally fails when it tries to read much more data than the max batch size allows. Tracing this back, it seems to come down to the "is_hvm" method of the XendConfig class, which checks the "HVM_boot_policy" parameter. For some reason, it seems this parameter is not being preserved during domain creation with xenapi, because by the time migration occurs, the dictionary item no longer exists. I''ll continue to dig further, but if anyone has any thoughts on this, I''d welcome pointers. Ben Guthro wrote:> I am seeing a problem when live migrating a guest under unstable when > I have xm configured to XenAPI. > > If my xm-config.xml file has the XenAPI stuff commented out - live > migrate (xm migrate --live <domid> <dest IP>) seems to be working in > my test environment. > > However, if I enable it - I get failures when migrating the same guest > (see dump from xend.log below.) They obviously go down quite > different code paths in these two cases. > > Being somewhat new to the xm, and xenapi control stacks - I was hoping > someone might point me in the right direction to track down what might > be happening here... > > Any help is appreciated. > > Ben > > Xend log snippet: > > 2007-11-07 09:01:35 15422] DEBUG (XendCheckpoint:88) [xc_save]: > /usr/lib64/xen/bin/xc_save 20 2 0 0 5 > [2007-11-07 09:01:35 15422] INFO (XendCheckpoint:362) Saving memory > pages: iter 1 0%ERROR Internal error: Error when writing to state > file (5) (errno 104) > [2007-11-07 09:01:35 15422] INFO (XendCheckpoint:362) Save exit rc=1 > [2007-11-07 09:01:36 15422] ERROR (XendCheckpoint:140) Save failed on > domain VS022 (2). > Traceback (most recent call last): > File "/usr/lib64/python/xen/xend/XendCheckpoint.py", line 108, in save > forkHelper(cmd, fd, saveInputHandler, False) > File "/usr/lib64/python/xen/xend/XendCheckpoint.py", line 350, in > forkHelper > raise XendError("%s failed" % string.join(cmd)) > XendError: /usr/lib64/xen/bin/xc_save 20 2 0 0 5 failed > [2007-11-07 09:01:36 15422] DEBUG (XendDomainInfo:1856) > XendDomainInfo.resumeDomain(2) > [2007-11-07 09:01:36 15422] ERROR (xmlrpclib2:166) Internal error > handling VM.migrate > Traceback (most recent call last): > File "/usr/lib64/python/xen/util/xmlrpclib2.py", line 131, in > _marshaled_dispatch > response = self._dispatch(method, params) > File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line 406, in _dispatch > return func(*params) > File "/usr/lib64/python/xen/xend/XendAPI.py", line 221, in f > return func(self, *args, **kwargs) > File "/usr/lib64/python/xen/xend/XendAPI.py", line 256, in check_session > return func(self, session, *args, **kwargs) > File "/usr/lib64/python/xen/xend/XendAPI.py", line 312, in <lambda> > ''VM'', func, *args, **kwargs) > File "/usr/lib64/python/xen/xend/XendAPI.py", line 268, in _check_ref > return func(api, session, ref, *args, **kwargs) > File "/usr/lib64/python/xen/xend/XendAPI.py", line 1763, in VM_migrate > bool(live), resource, port) > File "/usr/lib64/python/xen/xend/XendDomain.py", line 1282, in > domain_migrate > XendCheckpoint.save(sock.fileno(), dominfo, True, live, dst) > File "/usr/lib64/python/xen/xend/XendCheckpoint.py", line 142, in save > dominfo.resumeDomain() > File "/usr/lib64/python/xen/xend/XendDomainInfo.py", line 1885, in > resumeDomain > if self.is_hvm(): > File "/usr/lib64/python/xen/xend/XendDomainInfo.py", line 2139, in > __getattr__ > raise AttributeError() > AttributeError > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2007-Nov-08 16:22 UTC
Re: [Xen-devel] Re: Live Migration problems using xenapi with xm
Ben Guthro writes ("[Xen-devel] Re: Live Migration problems using xenapi
with xm"):> I''ve managed to track this down to being a bug in HVM migration
using
> XenAPI.
Aha.
> It seems that restore is trying to restore a PV guest, while save is
> trying to save an HVM guest... [ etc. ]
Thanks for investigating.
> I''ll continue to dig further, but if anyone has any thoughts on
this,
> I''d welcome pointers.
I don''t have any thoughts right now, I''m afraid. I hope to be
in a
better position to help out very shortly; There''s quite a lot of
review, testing and general sorting out of XenAPI needed which I''ll be
looking into.
In the meantime, good luck and do let us know what more you find.
Regards,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Ben Guthro
2007-Nov-08 16:36 UTC
Re: [Xen-devel] Re: Live Migration problems using xenapi with xm
Thanks for your reply, I''m wondering - is this uncharted territory, or has migration on HVM (with xenapi) worked in the past? We''ll be happy to help fix this...but it would go a long way to know what is thought to be working. Strangely, I have found that I can migrate guests with xenapi that were not created with xenapi. Just as a hack, I also tried always returning True for is_hvm. This seemed to get the migration process further, but seemed to get hung up on closing(or connecting) the vkbd device - leaving the domain in a broken state at both the sender, and receiver Ben Ian Jackson wrote:> Ben Guthro writes ("[Xen-devel] Re: Live Migration problems using xenapi with xm"): > >> I''ve managed to track this down to being a bug in HVM migration using >> XenAPI. >> > > Aha. > > >> It seems that restore is trying to restore a PV guest, while save is >> trying to save an HVM guest... [ etc. ] >> > > Thanks for investigating. > > >> I''ll continue to dig further, but if anyone has any thoughts on this, >> I''d welcome pointers. >> > > I don''t have any thoughts right now, I''m afraid. I hope to be in a > better position to help out very shortly; There''s quite a lot of > review, testing and general sorting out of XenAPI needed which I''ll be > looking into. > > In the meantime, good luck and do let us know what more you find. > > Regards, > Ian. >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jim Fehlig
2007-Nov-08 18:02 UTC
Re: [Xen-devel] Re: Live Migration problems using xenapi with xm
Ben Guthro wrote:> Thanks for your reply, > > I''m wondering - is this uncharted territory, or has migration on HVM > (with xenapi) worked in the past? We''ll be happy to help fix > this...but it would go a long way to know what is thought to be working.I would suspect you are in uncharted territory. Currently I don''t think there are a lot of xenapi users. In the past I have generally had to fix bugs or provide implementation for xenapi features previously not consumed by my app. You are perhaps the first to attempt migration (almost certainly of HVM guests) through xenapi. Regards, Jim _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2007-Nov-08 18:09 UTC
Re: [Xen-devel] Re: Live Migration problems using xenapi with xm
Jim Fehlig writes ("Re: [Xen-devel] Re: Live Migration problems using
xenapi with xm"):> I would suspect you are in uncharted territory. Currently I don''t
think
> there are a lot of xenapi users. In the past I have generally had to
> fix bugs or provide implementation for xenapi features previously not
> consumed by my app. You are perhaps the first to attempt migration
> (almost certainly of HVM guests) through xenapi.
Jim may well be right, I''m afraid. I''ve only just arrived so
to speak
but I think the implementation in xend certainly needs work.
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Ben Guthro
2007-Nov-08 20:18 UTC
Re: [Xen-devel] Re: Live Migration problems using xenapi with xm
I think I have hacked together something that seems to work... HVM Live migration seems to succeed if I do the following: a.) Ignore the fact that receive thinks it is a PV guest (force is_hvm to return True) b.) Modify XendDomainInfo.waitForDevices to ignore "vkbd" and "vfb" devices for HVM guests We don''t really do much PV guest stuff here, so I''m not real sure what the vfb, and vkbd are in the PV world... However, I''m not sure that they are necessary in HVM (though please correct me if I''m wrong) I''d like to come up with a patch for this, but would like to hear thoughts on what the right way to go about this is. Should I ignore those devices in waitForDevices? Should they not be there in the first place? Ian Jackson wrote:> Jim Fehlig writes ("Re: [Xen-devel] Re: Live Migration problems using xenapi with xm"): > >> I would suspect you are in uncharted territory. Currently I don''t think >> there are a lot of xenapi users. In the past I have generally had to >> fix bugs or provide implementation for xenapi features previously not >> consumed by my app. You are perhaps the first to attempt migration >> (almost certainly of HVM guests) through xenapi. >> > > Jim may well be right, I''m afraid. I''ve only just arrived so to speak > but I think the implementation in xend certainly needs work. > > Thanks, > Ian. >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel