xen discuss - Jun 2008 - SXCE b90 dom0 panic

I¹ve experienced another dom0 panic on my SXCE b90 xVM server, with perhaps
a more replicable scenario.

This time it appears that it was caused by ³virsh destroy² on a spinning
Ubuntu Hardy 8.0.4 domU running the vanilla Xen 3.1 2.6.18 domU kernel with
a full disk and a tight OOM killer loop.

Immediately after issuing the ³virsh destroy² command on the dom0
(lab-xvm-00-a), this panic appears to have occurred on the dom0.

As you can see at the top of this log, the xpvd devices were torn down and
it looks like some network traffic came in during the final stages of the
teardown and caused this panic.

Jun 25 15:09:42 lab-xvm-00-a genunix: [ID 408114 kern.info]
/xpvd/xdb@4,51712 (xdb5) offline
Jun 25 15:09:42 lab-xvm-00-a genunix: [ID 408114 kern.info] /xpvd/xnb@4,0
(xnbo3) offline
Jun 25 15:09:44 lab-xvm-00-a unix: [ID 836849 kern.notice]
Jun 25 15:09:44 lab-xvm-00-a ^Mpanic[cpu3]/thread=ffffff002d57bc80:
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 103648 kern.notice] mutex_enter:
bad mutex, lp=ffffff06de5ed130 owner=ffffff002de1ac80
thread=ffffff002d57bc80
Jun 25 15:09:44 lab-xvm-00-a unix: [ID 100000 kern.notice]
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d57b440 unix:mutex_panic+73 ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d57b4b0 unix:mutex_vector_enter+452 ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d57b580 xnb:xnb_copy_to_peer+53 ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d57b5b0 xnbo:xnbo_from_mac+1c ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d57b630 mac:mac_do_rx+b9 ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d57b660 mac:mac_rx+1b ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d57b6b0 vnic:vnic_rx+59 ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d57b760 vnic:vnic_promisc_rx+12e ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d57b7c0 vnic:vnic_classifier_rx+43 ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d57b840 mac:mac_do_rx+b9 ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d57b870 mac:mac_rx+1b ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d57b940 bnx:mm_indicate_rx+535 ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d57bb30 bnx:lm_service_rx_int+9b ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d57bb80 bnx:um_service_interrupt+d7 ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d57bbc0 bnx:bx2_intr+168 ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d57bc20 unix:av_dispatch_autovect+78 ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d57bc60 unix:dispatch_hardint+33 ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d5419d0 unix:switch_sp_and_call+13 ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d541a20 unix:do_interrupt+9b ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d541ad0 unix:xen_callback_handler+370 ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d541ae0 unix:xen_callback+cd ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d541be0 unix:HYPERVISOR_sched_op+29 ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d541bf0 unix:HYPERVISOR_block+11 ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d541c00 unix:mach_cpu_idle+12 ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d541c40 unix:cpu_idle+e1 ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d541c60 unix:idle+10e ()
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 655072 kern.notice]
ffffff002d541c70 unix:thread_start+8 ()
Jun 25 15:09:44 lab-xvm-00-a unix: [ID 100000 kern.notice]
Jun 25 15:09:44 lab-xvm-00-a genunix: [ID 672855 kern.notice] syncing file
systems...
Jun 25 15:09:45 lab-xvm-00-a genunix: [ID 733762 kern.notice]  5
Jun 25 15:09:46 lab-xvm-00-a genunix: [ID 733762 kern.notice]  1
Jun 25 15:09:47 lab-xvm-00-a genunix: [ID 904073 kern.notice]  done
Jun 25 15:09:48 lab-xvm-00-a genunix: [ID 111219 kern.notice] dumping to
/dev/dsk/c1t0d0s3, offset 431030272
, content: kernel
Jun 25 15:09:56 lab-xvm-00-a genunix: [ID 409368 kern.notice] ^M100% done:
323057 pages dumped, compression
ratio 5.62,
Jun 25 15:09:56 lab-xvm-00-a genunix: [ID 851671 kern.notice] dump succeeded
Jun 25 15:12:05 lab-xvm-00-a genunix: [ID 540533 kern.notice] ^MSunOS
Release 5.11 Version snv_90 64-bit
Jun 25 15:12:05 lab-xvm-00-a genunix: [ID 172908 kern.notice] Copyright
1983-2008 Sun Microsystems, Inc.  All rights reserved.
...
Jun 25 15:12:20 lab-xvm-00-a savecore: [ID 570001 auth.error] reboot after
panic: mutex_enter: bad mutex, lp
=ffffff06de5ed130 owner=ffffff002de1ac80 thread=ffffff002d57bc80
Jun 25 15:12:20 lab-xvm-00-a savecore: [ID 748169 auth.error] saving system
crash dump in /var/crash/lab-xvm
-00-a/*.3

I can provide the crashdump or any other pertinent data upon request.

 - Ian C. Blenke <iblenke@csdvrscom> <ian@blenke.com>

On 25 Jun 2008, at 4:58pm, Ian Blenke wrote:
> I’ve experienced another dom0 panic on my SXCE b90 xVM server, with  
> perhaps a more replicable scenario.
>
> This time it appears that it was caused by “virsh destroy” on a  
> spinning Ubuntu Hardy 8.0.4 domU running the vanilla Xen 3.1 2.6.18  
> domU kernel with a full disk and a tight OOM killer loop.
>
> Immediately after issuing the “virsh destroy” command on the dom0  
> (lab-xvm-00-a), this panic appears to have occurred on the dom0.
This looks to be the same as Jeffrey Kuhn''s problem that Max is  
looking at.

Idle curiosity, could you show us the output of ''syseventadm
list''
please?

On 6/25/08 12:11 PM, "David Edmondson" <dme@sun.com> wrote:
> syseventadm list
Sure thing:

# /usr/sbin/syseventadm list
vendor=SUNW publisher=pcie_pci class=EC_dr subclass=ESC_dr_req
/usr/lib/pci/pcidr class=$class subclass=$subclass
publisher=$publisher dr_request_type=$dr_request_type dr_ap_id=$dr_ap_id
vendor=SUNW publisher=px_pci class=EC_dr subclass=ESC_dr_req
/usr/lib/pci/pcidr class=$class subclass=$subclass publisher=$publisher
dr_request_type=$dr_request_type dr_ap_id=$dr_ap_id vendor=SUNW
publisher=pxb_bcm class=EC_dr subclass=ESC_dr_req /usr/lib/pci/pcidr
class=$class subclass=$subclass publisher=$publisher
dr_request_type=$dr_request_type dr_ap_id=$dr_ap_id vendor=SUNW
publisher=pxb_plx class=EC_dr subclass=ESC_dr_req /usr/lib/pci/pcidr
class=$class subclass=$subclass publisher=$publisher
dr_request_type=$dr_request_type dr_ap_id=$dr_ap_id class=EC_xendev
/usr/lib/xen/scripts/xpvd-event action=$subclass domain=$domain vdev=$vdev
device=$device devclass=$devclass febe=$fob class=EC_xpvsys
/usr/lib/xen/scripts/xpvsys-event subclass=$subclass shutdown=$shutdown

Don''t hesitate to ask if there''s anything else I can provide
to help hunt
this down.

 - Ian C. Blenke <iblenke@csdvrs.com> <ian@blenke.com>

Can you send us the log file: /var/log/xen/xpvd-event.log?
Are you able to easily reproduce it?

Thanks,
Max

Ian Blenke wrote:
> On 6/25/08 12:11 PM, "David Edmondson" <dme@sun.com> wrote:
>
>   
>> syseventadm list
>>     
>
> Sure thing:
>
> # /usr/sbin/syseventadm list
> vendor=SUNW publisher=pcie_pci class=EC_dr subclass=ESC_dr_req
> /usr/lib/pci/pcidr class=$class subclass=$subclass
> publisher=$publisher dr_request_type=$dr_request_type dr_ap_id=$dr_ap_id
> vendor=SUNW publisher=px_pci class=EC_dr subclass=ESC_dr_req
> /usr/lib/pci/pcidr class=$class subclass=$subclass publisher=$publisher
> dr_request_type=$dr_request_type dr_ap_id=$dr_ap_id vendor=SUNW
> publisher=pxb_bcm class=EC_dr subclass=ESC_dr_req /usr/lib/pci/pcidr
> class=$class subclass=$subclass publisher=$publisher
> dr_request_type=$dr_request_type dr_ap_id=$dr_ap_id vendor=SUNW
> publisher=pxb_plx class=EC_dr subclass=ESC_dr_req /usr/lib/pci/pcidr
> class=$class subclass=$subclass publisher=$publisher
> dr_request_type=$dr_request_type dr_ap_id=$dr_ap_id class=EC_xendev
> /usr/lib/xen/scripts/xpvd-event action=$subclass domain=$domain vdev=$vdev
> device=$device devclass=$devclass febe=$fob class=EC_xpvsys
> /usr/lib/xen/scripts/xpvsys-event subclass=$subclass shutdown=$shutdown
>
> Don''t hesitate to ask if there''s anything else I can
provide to help hunt
> this down.
>
>  - Ian C. Blenke <iblenke@csdvrs.com> <ian@blenke.com>
>
>
> _______________________________________________
> xen-discuss mailing list
> xen-discuss@opensolaris.org
>