Xen users - Mar 2010 - XCP VM restart strangeness

Just when I think I''ve got XCP beat into submission, it sneaks up and 
whacks me again :).

I rebooted a CentOS VM from it''s command line and it failed to restart.
A check of vm-list showed that the VM was showing halted on the pool 
master and the slaves, but trying to vm-start resulted in an error that 
the session reference was invalid.

After a great deal of fooling with it, I managed to clear the problem by 
doing a vm-reset-powerstate, which I wouldn''t have expected to make any
difference, since the VM was already showing halted.

I toss this out for informational purposes, in case someone else runs 
into this :).

-- 
Vern Burke

SwiftWater Telecom
http://www.swiftwatertel.com
ISP/CLEC Engineering Services
Data Center Services
Remote Backup Services

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi Vern,

Those are very odd symptoms! It''s good that you''ve got a
workaround.

If this happens again, grab a ''bugtool'' on the master host
when it''s in this state:
  xen-bugtool --silent --yestoall
If you send me the output then I''ll do some digging to see if I can
figure out what happened.

Cheers,
Dave
> -----Original Message-----
> From: SwiftWater Telecom Customer Service
> [mailto:service@swiftwatertel.com]
> Sent: 09 March 2010 20:19
> To: Dave Scott; xen-users@lists.xensource.com
> Subject: XCP VM restart strangeness
> 
> Just when I think I''ve got XCP beat into submission, it sneaks up
and
> whacks me again :).
> 
> I rebooted a CentOS VM from it''s command line and it failed to
restart.
> A check of vm-list showed that the VM was showing halted on the pool
> master and the slaves, but trying to vm-start resulted in an error that
> the session reference was invalid.
> 
> After a great deal of fooling with it, I managed to clear the problem
> by
> doing a vm-reset-powerstate, which I wouldn''t have expected to
make any
> difference, since the VM was already showing halted.
> 
> I toss this out for informational purposes, in case someone else runs
> into this :).
> 
> --
> Vern Burke
> 
> SwiftWater Telecom
> http://www.swiftwatertel.com
> ISP/CLEC Engineering Services
> Data Center Services
> Remote Backup Services
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

This is certainly odd. I was just testing my watchdog (it automatically 
restarts halted VMs that are marked to be restarted if they''re found 
halted using xe), ran it from the Linux command line on my command and 
control server with a halted VM for it to restart and got:

[root@xc1-me1-vps007 cgi-bin]# sh ./watchdog.cgi
You gave an invalid session reference.  It may have been invalidated by 
a server restart, or timed out.  You should get a new session handle, 
using one of the session.login_ calls.  This error does not invalidate 
the current connection.  The handle parameter echoes the bad value given.
handle: OpaqueRef:90c823fb-548e-93f9-b2d2-34809fe74e87

I hopped on the same server with a new SSH connection, reran the 
watchdog and it went fine and restarted the down VM perfectly. I killed 
the VM, went back to the original term window, reran the watchdog, and 
it ran perfectly.

I swear this place is built over an ancient Indian burial ground.


Vern Burke

SwiftWater Telecom
http://www.swiftwatertel.com
ISP/CLEC Engineering Services
Data Center Services
Remote Backup Services

On 3/9/2010 3:41 PM, Dave Scott wrote:
> Hi Vern,
>
> Those are very odd symptoms! It''s good that you''ve got a
workaround.
>
> If this happens again, grab a ''bugtool'' on the master
host when it''s in this state:
>    xen-bugtool --silent --yestoall
> If you send me the output then I''ll do some digging to see if I
can figure out what happened.
>
> Cheers,
> Dave
>
>> -----Original Message-----
>> From: SwiftWater Telecom Customer Service
>> [mailto:service@swiftwatertel.com]
>> Sent: 09 March 2010 20:19
>> To: Dave Scott; xen-users@lists.xensource.com
>> Subject: XCP VM restart strangeness
>>
>> Just when I think I''ve got XCP beat into submission, it sneaks
up and
>> whacks me again :).
>>
>> I rebooted a CentOS VM from it''s command line and it failed to
restart.
>> A check of vm-list showed that the VM was showing halted on the pool
>> master and the slaves, but trying to vm-start resulted in an error that
>> the session reference was invalid.
>>
>> After a great deal of fooling with it, I managed to clear the problem
>> by
>> doing a vm-reset-powerstate, which I wouldn''t have expected to
make any
>> difference, since the VM was already showing halted.
>>
>> I toss this out for informational purposes, in case someone else runs
>> into this :).
>>
>> --
>> Vern Burke
>>
>> SwiftWater Telecom
>> http://www.swiftwatertel.com
>> ISP/CLEC Engineering Services
>> Data Center Services
>> Remote Backup Services
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users