CentOS virt - Mar 2010 - reboot guest on panic

I have a guest that keeps crashing and want to automatically reboot it
when it crashes. See:

  xen PV guest kernel 2.6.32 processes lock up in D state
  https://bugzilla.redhat.com/show_bug.cgi?id=550724

if you want to look at the details on the crashing.

Anyway, I boot the guest with the kernel command line parameter:

  hung_task_panic=1

I have kernel.panic = 15 in the guest /etc/sysctl.conf

In the guest config file in dom0 I have:

  on_poweroff = "destroy"
  on_reboot = "restart"
  on_crash = "restart"

The guest manages to panic when it detects the hung tasks and I get this
on the guest console:

  Kernel panic - not syncing: hung_task: blocked tasks
  Rebooting in 15 seconds..

However, it never restarts. It just hangs around until I do a

  xm destroy <guest>
  xm create <guest>

Have I missed something?

-- 
Norman Gaywood, Computer Systems Officer
University of New England, Armidale, NSW 2351, Australia

ngaywood at une.edu.au            Phone: +61 (0)2 6773 3337
http://mcs.une.edu.au/~norm    Fax:   +61 (0)2 6773 3312

Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html

----- "Norman Gaywood" <ngaywood at une.edu.au> wrote:
> I have a guest that keeps crashing and want to automatically reboot
> it when it crashes. See:
> 
>   xen PV guest kernel 2.6.32 processes lock up in D state
>   https://bugzilla.redhat.com/show_bug.cgi?id=550724
> 
> if you want to look at the details on the crashing.
If your guest's state is corrupted, you can't rely on its behavior. For
that reason, you should set up a watchdog and only rely on the panic behavior as
a preliminary measure.

-- 
Christopher G. Stach II
http://ldsys.net/~cgs/

----- "Norman Gaywood" <ngaywood at une.edu.au> wrote:
> > If your guest's state is corrupted, you can't rely on its
behavior.
> For that reason, you should set up a watchdog and only rely on the
> panic behavior as a preliminary measure.
>  
> I figured that I had setup a watchdog with the hung_task_panic=1 guest
> parameter. Is there another way of setting up a watchdog in this case?
Internal watchdogs are seldom as effective as you need them to be. I meant an
external one. There was talk in the past about a Xen watchdog driver, but I
don't know where that ended up. (I use existing code that works for me, so I
haven't kept up.)

There are various ways to set one up. You can do it on your own with a couple of
scripts and xenstore, if you're so inclined. If not, there are a lot of
existing examples and projects that will handle it for you. This one that uses
QEMU and unfortunately only KVM is decent and should give you enough ideas on
how to implement one: http://rwmj.wordpress.com/2010/03/03/what-is-a-watchdog/

-- 
Christopher G. Stach II
http://ldsys.net/~cgs/

On Sun, Mar 14, 2010 at 09:03:23AM +1100, Norman Gaywood
wrote:
> I have a guest that keeps crashing and want to automatically reboot it
> when it crashes. See:
> 
>   xen PV guest kernel 2.6.32 processes lock up in D state
>   https://bugzilla.redhat.com/show_bug.cgi?id=550724
> 
> if you want to look at the details on the crashing.
> 
Btw please see: 
http://wiki.xensource.com/xenwiki/XenCommonProblems

Especially the chapter about debugging crashed guests.
It would be very helpful to grab a stacktrace of the crashed guest to debug it.

So set on_crash=preserve for the guest, and then use xenctx (with the guest
kernel System.map)
to get the stack trace..

Redhat bugzilla seems to be down, so I can't check the details about 
the bugreport. Is the guest single-vcpu or multi-vcpu? 32bit or 64bit?

-- Pasi
> Anyway, I boot the guest with the kernel command line parameter:
> 
>   hung_task_panic=1
> 
> I have kernel.panic = 15 in the guest /etc/sysctl.conf
> 
> In the guest config file in dom0 I have:
> 
>   on_poweroff = "destroy"
>   on_reboot = "restart"
>   on_crash = "restart"
> 
> The guest manages to panic when it detects the hung tasks and I get this
> on the guest console:
> 
>   Kernel panic - not syncing: hung_task: blocked tasks
>   Rebooting in 15 seconds..
> 
> However, it never restarts. It just hangs around until I do a
> 
>   xm destroy <guest>
>   xm create <guest>
> 
> Have I missed something?
> 
> -- 
> Norman Gaywood, Computer Systems Officer
> University of New England, Armidale, NSW 2351, Australia
> 
> ngaywood at une.edu.au            Phone: +61 (0)2 6773 3337
> http://mcs.une.edu.au/~norm    Fax:   +61 (0)2 6773 3312
> 
> Please avoid sending me Word or PowerPoint attachments.
> See http://www.gnu.org/philosophy/no-word-attachments.html
> _______________________________________________
> CentOS-virt mailing list
> CentOS-virt at centos.org
> http://lists.centos.org/mailman/listinfo/centos-virt

Try this in the guest:

echo "1" > /proc/sys/kernel/panic_on_oops
echo "5" > /proc/sys/kernel/panic

On 03/13/2010 05:03 PM, Norman Gaywood wrote:
> I have a guest that keeps crashing and want to automatically reboot it
> when it crashes. See:
>
>    xen PV guest kernel 2.6.32 processes lock up in D state
>    https://bugzilla.redhat.com/show_bug.cgi?id=550724
>
> if you want to look at the details on the crashing.
>
> Anyway, I boot the guest with the kernel command line parameter:
>
>    hung_task_panic=1
>
> I have kernel.panic = 15 in the guest /etc/sysctl.conf
>
> In the guest config file in dom0 I have:
>
>    on_poweroff = "destroy"
>    on_reboot = "restart"
>    on_crash = "restart"
>
> The guest manages to panic when it detects the hung tasks and I get this
> on the guest console:
>
>    Kernel panic - not syncing: hung_task: blocked tasks
>    Rebooting in 15 seconds..
>
> However, it never restarts. It just hangs around until I do a
>
>    xm destroy<guest>
>    xm create<guest>
>
> Have I missed something?
>
>

----- "Norman Gaywood" <ngaywood at une.edu.au> wrote:
> Well yes, seems I've hit trouble this time. Was expecting some trouble
> with the desktop apps and was happy to deal with that.  I was not
> expecting significant kernel trouble however.
It's Fedora. It's *always* broken.
> Rebuilding a modern desktop distro with an older kernel is a lot of
> work.
It's not that bad. The only things you really need to match up are glibc and
the kernel. Beyond that, most things are reasonably source compatible with a
slightly older glibc. I don't think you would need to go that far, though.
What do you need in Fedora that you can't get in the almost equally
substandard desktop experience from CentOS or any other enterprise distribution?
(I use CentOS as my primary desktop.)
> That's what distributions are for right?
Gentoo? :)

-- 
Christopher G. Stach II
http://ldsys.net/~cgs/