Schober Walter
2007-May-18 10:07 UTC
[Xen-users] How do I find out why Dom0 crashes on guest startup?
Before Wednesday all run fine. Then I must have done something - I tried to get rid of those IPMI error messages on starting hpasm, installed the latest RPMs from HP website - but can''t tell exactly when it started that the DomU startup crashed my Dom0. Anyway ... How can I find out now, what exactly causes my Dom0 to crash? Which logs would tell me that? /var/log/messages: Sometimes xenbr changes into forwarding of tap0 before, sometimes not. /var/log/xen/xend.log: The only thing that sounds strange: [2007-05-18 11:56:58 xend 3475] DEBUG (XendDomain:153) number of vcpus to use is 0 Meanwhile: hp-OpenIMPI, hpasm, hprsm, cmanic removed, OpenIPMI removed. Still, the Dom0 crashed (just reboots, no output on Display, SSH connection interrupted) on starting a DomU. System: Centos 5 x86_64 @ HP DL360G5, latest yum update done. Will attach more info on config, when needed. Many thanks! Walter _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Petersson, Mats
2007-May-18 10:12 UTC
RE: [Xen-users] How do I find out why Dom0 crashes on guest startup?
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > Schober Walter > Sent: 18 May 2007 11:08 > To: xen-users@lists.xensource.com > Subject: [Xen-users] How do I find out why Dom0 crashes on > guest startup? > > Before Wednesday all run fine. Then I must have done > something - I tried to get rid of those IPMI error messages > on starting hpasm, installed the latest RPMs from HP website > - but can''t tell exactly when it started that the DomU > startup crashed my Dom0. > > Anyway ... How can I find out now, what exactly causes my Dom0 > to crash? Which logs would tell me that? > > /var/log/messages: Sometimes xenbr changes into forwarding of > tap0 before, sometimes not. > /var/log/xen/xend.log: The only thing that sounds strange: > [2007-05-18 11:56:58 xend 3475] DEBUG (XendDomain:153) number > of vcpus to use is 0 > > Meanwhile: hp-OpenIMPI, hpasm, hprsm, cmanic removed, > OpenIPMI removed. > > Still, the Dom0 crashed (just reboots, no output on Display, > SSH connection interrupted) on starting a DomU.Are you able to get serial output from the server to another machine? If so, set xen to use "console=com1 com1=115200,8n1". If not, try adding "noreboot". [Both of these are arguments on the "xen.gz" line in your /boot/grub/grub.conf] -- Mats> > System: Centos 5 x86_64 @ HP DL360G5, latest yum update done. > > Will attach more info on config, when needed. > > Many thanks! > Walter > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Schober Walter
2007-May-18 10:35 UTC
RE: [Xen-users] How do I find out why Dom0 crashes on guest startup?
Thanks Mats! I can do a remote serial console via iLO2 on com2: but cannot cut&paste from it (Hello HP!) - so I have screenshots only (can send them on request) ... Finally it tells me Panic on CPU 2: CPU2 FATAL PAGE FAULT [error_code=0000] Faulting linear address: 000000000000000 It happens on the (XEN) (GUEST: 1) Starting emulated 16-bit real-mode: ip=0600:000 Stack Trace can be sent on request as JPG. br Walter -----Original Message----- From: Petersson, Mats [mailto:Mats.Petersson@amd.com] Sent: Friday, May 18, 2007 12:12 PM To: Schober Walter; xen-users@lists.xensource.com Subject: RE: [Xen-users] How do I find out why Dom0 crashes on guest startup?> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > Schober Walter > Sent: 18 May 2007 11:08 > To: xen-users@lists.xensource.com > Subject: [Xen-users] How do I find out why Dom0 crashes on > guest startup? > > Before Wednesday all run fine. Then I must have done > something - I tried to get rid of those IPMI error messages > on starting hpasm, installed the latest RPMs from HP website > - but can''t tell exactly when it started that the DomU > startup crashed my Dom0. > > Anyway ... How can I find out now, what exactly causes my Dom0 > to crash? Which logs would tell me that? > > /var/log/messages: Sometimes xenbr changes into forwarding of > tap0 before, sometimes not. > /var/log/xen/xend.log: The only thing that sounds strange: > [2007-05-18 11:56:58 xend 3475] DEBUG (XendDomain:153) number > of vcpus to use is 0 > > Meanwhile: hp-OpenIMPI, hpasm, hprsm, cmanic removed, > OpenIPMI removed. > > Still, the Dom0 crashed (just reboots, no output on Display, > SSH connection interrupted) on starting a DomU.Are you able to get serial output from the server to another machine? If so, set xen to use "console=com1 com1=115200,8n1". If not, try adding "noreboot". [Both of these are arguments on the "xen.gz" line in your /boot/grub/grub.conf] -- Mats> > System: Centos 5 x86_64 @ HP DL360G5, latest yum update done. > > Will attach more info on config, when needed. > > Many thanks! > Walter > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Petersson, Mats
2007-May-18 10:36 UTC
RE: [Xen-users] How do I find out why Dom0 crashes on guest startup?
> -----Original Message----- > From: Schober Walter [mailto:Walter.Schober@neotel.at] > Sent: 18 May 2007 11:35 > To: xen-users@lists.xensource.com > Cc: Petersson, Mats > Subject: RE: [Xen-users] How do I find out why Dom0 crashes > on guest startup? > > Thanks Mats! I can do a remote serial console via iLO2 on com2: but > cannot cut&paste from it (Hello HP!) - so I have screenshots only (can > send them on request) ... > > Finally it tells me > Panic on CPU 2: > CPU2 FATAL PAGE FAULT > [error_code=0000] > Faulting linear address: 000000000000000 > > It happens on the > (XEN) (GUEST: 1) Starting emulated 16-bit real-mode: ip=0600:000 > > Stack Trace can be sent on request as JPG.I think a stack-trace would be useful here. I suspect its'' someplace in the HVM/VMX (or SVM) code, but it''s much easier to tell what''s going on from a stack-trace. -- Mats> > br > Walter > > -----Original Message----- > From: Petersson, Mats [mailto:Mats.Petersson@amd.com] > Sent: Friday, May 18, 2007 12:12 PM > To: Schober Walter; xen-users@lists.xensource.com > Subject: RE: [Xen-users] How do I find out why Dom0 crashes on guest > startup? > > > > > -----Original Message----- > > From: xen-users-bounces@lists.xensource.com > > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > > Schober Walter > > Sent: 18 May 2007 11:08 > > To: xen-users@lists.xensource.com > > Subject: [Xen-users] How do I find out why Dom0 crashes on > > guest startup? > > > > Before Wednesday all run fine. Then I must have done > > something - I tried to get rid of those IPMI error messages > > on starting hpasm, installed the latest RPMs from HP website > > - but can''t tell exactly when it started that the DomU > > startup crashed my Dom0. > > > > Anyway ... How can I find out now, what exactly causes my Dom0 > > to crash? Which logs would tell me that? > > > > /var/log/messages: Sometimes xenbr changes into forwarding of > > tap0 before, sometimes not. > > /var/log/xen/xend.log: The only thing that sounds strange: > > [2007-05-18 11:56:58 xend 3475] DEBUG (XendDomain:153) number > > of vcpus to use is 0 > > > > Meanwhile: hp-OpenIMPI, hpasm, hprsm, cmanic removed, > > OpenIPMI removed. > > > > Still, the Dom0 crashed (just reboots, no output on Display, > > SSH connection interrupted) on starting a DomU. > > Are you able to get serial output from the server to another machine? > > If so, set xen to use "console=com1 com1=115200,8n1". > > If not, try adding "noreboot". > > [Both of these are arguments on the "xen.gz" line in your > /boot/grub/grub.conf] > > -- > Mats > > > > System: Centos 5 x86_64 @ HP DL360G5, latest yum update done. > > > > Will attach more info on config, when needed. > > > > Many thanks! > > Walter > > > > > > > > > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Schober Walter
2007-May-18 10:46 UTC
RE: [Xen-users] How do I find out why Dom0 crashes on guest startup?
I managed it via SSH to iLO2 :-) Console output, guest config below:
(XEN) (file=hvm.c, line=197) Allocated port 3 for hvm.
(XEN) (file=hvm.c, line=197) Allocated port 4 for hvm.
(XEN) (file=hvm.c, line=197) Allocated port 5 for hvm.
(XEN) (file=hvm.c, line=197) Allocated port 6 for hvm.
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c23
(pseudophys a0): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c22
(pseudophys a1): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c21
(pseudophys a2): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c20
(pseudophys a3): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c1f
(pseudophys a4): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c1e
(pseudophys a5): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c1d
(pseudophys a6): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c1c
(pseudophys a7): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c1b
(pseudophys a8): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c1a
(pseudophys a9): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c19
(pseudophys aa): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c18
(pseudophys ab): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c17
(pseudophys ac): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c16
(pseudophys ad): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c15
(pseudophys ae): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c14
(pseudophys af): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c13
(pseudophys b0): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c12
(pseudophys b1): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c11
(pseudophys b2): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c10
(pseudophys b3): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c0f
(pseudophys b4): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c0e
(pseudophys b5): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c0d
(pseudophys b6): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c0c
(pseudophys b7): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c0b
(pseudophys b8): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c0a
(pseudophys b9): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c09
(pseudophys ba): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c08
(pseudophys bb): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c07
(pseudophys bc): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c06
(pseudophys bd): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c05
(pseudophys be): count=2 type=0
(XEN) (file=memory.c, line=188) Dom1 freeing in-use page c8c04
(pseudophys bf): count=2 type=0
(XEN) vmx_do_launch(): GUEST_CR3<=0027f7a0, HOST_CR3<=4f2bf000
(XEN) (GUEST: 1) HVM Loader
(XEN) (GUEST: 1) Detected Xen v3.0.3-rc5-8.1.3.el
(XEN) (GUEST: 1) Writing SMBIOS tables ...
(XEN) (GUEST: 1) Loading ROMBIOS ...
(XEN) (GUEST: 1) Creating MP tables ...
(XEN) (GUEST: 1) Loading Cirrus VGABIOS ...
(XEN) (GUEST: 1) Loading ACPI ...
(XEN) (GUEST: 1) Loading VMXAssist ...
(XEN) (GUEST: 1) VMX go ...
(XEN) (GUEST: 1) VMXAssist (Apr 9 2007)
(XEN) (GUEST: 1) Memory size 2000 MB
(XEN) (GUEST: 1) E820 map:
(XEN) (GUEST: 1) 0000000000000000 - 000000000009F000 (RAM)
(XEN) (GUEST: 1) 000000000009F000 - 00000000000A0000 (Reserved)
(XEN) (GUEST: 1) 00000000000A0000 - 00000000000C0000 (Type 16)
(XEN) (GUEST: 1) 00000000000F0000 - 0000000000100000 (Reserved)
(XEN) (GUEST: 1) 0000000000100000 - 000000007CFF0000 (RAM)
(XEN) (GUEST: 1) 000000007CFF0000 - 000000007CFFA000 (ACPI Data)
(XEN) (GUEST: 1) 000000007CFFA000 - 000000007CFFD000 (ACPI NVS)
(XEN) (GUEST: 1) 000000007CFFD000 - 000000007CFFE000 (Type 19)
(XEN) (GUEST: 1) 000000007CFFE000 - 000000007CFFF000 (Type 18)
(XEN) (GUEST: 1) 000000007CFFF000 - 000000007D000000 (Type 17)
(XEN) (GUEST: 1) 00000000FEC00000 - 0000000100000000 (Type 16)
(XEN) (GUEST: 1)
(XEN) (GUEST: 1) Start BIOS ...
(XEN) (GUEST: 1) Starting emulated 16-bit real-mode: ip=F000:FFF0
(XEN) (GUEST: 1) rombios.c,v 1.138 2005/05/07 15:55:26 vruppert Exp $
(XEN) (GUEST: 1) Remapping master: ICW2 0x8 -> 0x20
(XEN) (GUEST: 1) Remapping slave: ICW2 0x70 -> 0x28
(XEN) (GUEST: 1) VGABios $Id: vgabios.c,v 1.61 2005/05/24 16:50:50
vruppert Exp $
(XEN) (GUEST: 1) HVMAssist BIOS, 1 cpu, $Revision: 1.138 $ $Date:
2005/05/07 15:55:26 $
(XEN) (GUEST: 1)
(XEN) (GUEST: 1) ata0-0: PCHS=16383/16/63 translation=lba
LCHS=1024/255/63
(XEN) (GUEST: 1) ata0 master: QEMU HARDDISK ATA-7 Hard-Disk (15360
MBytes)
(XEN) (GUEST: 1) ata0 slave: Unknown device
(XEN) (GUEST: 1)
(XEN) (GUEST: 1) Booting from Hard Disk...
(XEN) (GUEST: 1) int13_harddisk: function 41, unmapped device for
ELDL=81
(XEN) (GUEST: 1) int13_harddisk: function 08, unmapped device for
ELDL=81
(XEN) (GUEST: 1) *** int 15h function AX=00C0, BX=0000 not yet
supported!
(XEN) (GUEST: 1) int13_harddisk: function 15, unmapped device for
ELDL=81
(XEN) (GUEST: 1) *** int 15h function AX=EC00, BX=0002 not yet
supported!
(XEN) (GUEST: 1) KBD: unsupported int 16h function 03
(XEN) (GUEST: 1) int13_harddisk: function 15, unmapped device for
ELDL=81
(XEN) (GUEST: 1) int13_harddisk: function 02, unmapped device for
ELDL=81
(XEN) (GUEST: 1) int13_harddisk: function 41, unmapped device for
ELDL=81
(XEN) This hvm_vlapic is for P4, no work for De-assert init
(XEN) (file=hvm.c, line=665) AP 1 bringup suceeded.
(XEN) vmx_do_launch(): GUEST_CR3<=0027d7a0, HOST_CR3<=4f28e000
(XEN) (GUEST: 1) Start AP 1 from 00006000 ...
(XEN) (GUEST: 1) Starting emulated 16-bit real-mode: ip=0600:0000
(XEN) This hvm_vlapic is for P4, no work for De-assert init
(XEN) (file=hvm.c, line=665) AP 2 bringup suceeded.
(XEN) vmx_do_launch(): GUEST_CR3<=0027b7a0, HOST_CR3<=4f28a000
(XEN) (GUEST: 1) Start AP 2 from 00006000 ...
(XEN) (GUEST: 1) Starting emulated 16-bit real-mode: ip=0600:0000
(XEN) This hvm_vlapic is for P4, no work for De-assert init
(XEN) (file=hvm.c, line=665) AP 3 bringup suceeded.
(XEN) malloc vlapic regs error for vcpu 3
(XEN) vmx_do_launch(): GUEST_CR3<=002797a0, HOST_CR3<=4f289000
(XEN) (GUEST: 1) Start AP 3 from 00006000 ...
(XEN) (GUEST: 1) Starting emulated 16-bit real-mode: ip=0600:0000
(XEN) ----[ Xen-3.0.3-rc5-8.1.3.el5 x86_64 debug=n Not tainted ]----
(XEN) CPU: 2
(XEN) RIP: e010:[<ffff830000142da7>] vlapic_range+0x7/0x30
(XEN) RFLAGS: 0000000000010202 CONTEXT: hypervisor
(XEN) rax: 0000000000000001 rbx: ffff830000195b60 rcx:
0000000000000020
(XEN) rdx: 0000000000000000 rsi: 00000000fee00020 rdi:
ffff830000278080
(XEN) rbp: ffff8300001967e0 rsp: ffff83000021bae0 r8:
0000000000000000
(XEN) r9: 0000000000000001 r10: 0000000000000004 r11:
0000000000000000
(XEN) r12: ffff830000278080 r13: ffff8300110b30b0 r14:
0000000000000000
(XEN) r15: 0000000000000001 cr0: 000000008005003b cr4:
00000000000026f0
(XEN) cr3: 000000004f287000 cr2: 0000000000000000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e010
(XEN) Xen stack trace from rsp=ffff83000021bae0:
(XEN) ffff83000013e8b2 00000000c8abe067 0000000000000000
0000000000000000
(XEN) ffff83000021bc58 0000000000000007 ffffffff80072f1a
0000000000000000
(XEN) ffff8300110b30b0 ffff8300110b30a8 ffff830000278080
0000000000000004
(XEN) 0000000000000000 0000000000000001 ffff8300001402e1
00000000000c8ac2
(XEN) 00000000000c8ac0 00000001000c8abe 0000000000000000
00000000fee00020
(XEN) 01ff83000013b8c0 ffff830000278080 ffff83000021bc38
0000000000000001
(XEN) 00000000fee00020 0000000000000000 ffff83000021bc38
00000000fee00020
(XEN) ffff8300001409d3 0000000000000000 0000000000000001
0000000000000007
(XEN) ffff8300002796b0 0000000000000007 ffff830000141a1d
0000000000000000
(XEN) ffff830000166115 0000000000000000 ffff83000021bf28
ffff83000021bf28
(XEN) ffffffffff5fd020 ffff830000279690 ffff830000278080
0000000000040206
(XEN) 0000000000000000 04000002fee00000 0000000004060001
00ff5fd02025348b
(XEN) 0000000000000000 0000000000000000 00000000000fee00
00000000fee00762
(XEN) ffff830000278080 ffff830000fdc080 00000000fee00020
00000000fee00000
(XEN) 0000000000000001 000000000004f295 ffff830000159959
ffffffffff5fd020
(XEN) ffff81ffffffafe8 000000010000604c ffff8140ffffffd0
000000000004f296
(XEN) 000000000004f2a5 000000000004f2a8 ffffffff00000008
0000c09b00000000
(XEN) 000000000004f289 ffff830000278080 ffff830000278080
0000000000000001
(XEN) 00000000a0000003 000000000004f287 ffff83000015aa90
0000000000000282
(XEN) 000000000004f289 ffff830000278080 ffff8300001522d8
0000008b000d70c4
(XEN) Xen call trace:
(XEN) [<ffff830000142da7>] vlapic_range+0x7/0x30
(XEN) [<ffff83000013e8b2>] hvm_mmio_intercept+0x62/0x350
(XEN) [<ffff8300001402e1>] send_mmio_req+0x1d1/0x250
(XEN) [<ffff8300001409d3>] mmio_operands+0x103/0x150
(XEN) [<ffff830000141a1d>] handle_mmio+0xffd/0x12c0
(XEN) [<ffff830000166115>] handle_exception+0x45/0x6e
(XEN) [<ffff830000159959>]
sh_page_fault__shadow_4_guest_4+0x1019/0x1080
(XEN) [<ffff83000015aa90>]
sh_destroy_monitor_table__shadow_3_guest_3+0x40/0x60
(XEN) [<ffff8300001522d8>] sh_update_paging_modes+0x3e8/0x450
(XEN) [<ffff830000159d19>]
sh_update_cr3__shadow_4_guest_4+0x299/0x360
(XEN) [<ffff83000014f317>] vmx_set_cr0+0x4b7/0x950
(XEN) [<ffff83000012f77a>] smp_send_event_check_mask+0x3a/0x40
(XEN) [<ffff830000152130>] sh_update_paging_modes+0x240/0x450
(XEN) [<ffff83000014c6a7>] vmx_vmexit_do_cpuid+0x107/0x270
(XEN) [<ffff83000013b1f1>] hvm_do_resume+0x121/0x1a0
(XEN) [<ffff83000014c0fb>] vmx_do_page_fault+0x2b/0x50
(XEN) [<ffff830000150c09>] vmx_vmexit_handler+0x339/0xf00
(XEN) [<ffff830000142e6a>] cpu_has_pending_irq+0x2a/0x50
(XEN) [<ffff83000014a7a8>] vmx_intr_assist+0xf8/0x400
(XEN) [<ffff8300001517f8>] vmx_asm_vmexit_handler+0x28/0x30
(XEN)
(XEN) Pagetable walk from 0000000000000000:
(XEN) L4[0x000] = 0000000000000000 ffffffffffffffff
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 2:
(XEN) CPU2 FATAL PAGE FAULT
(XEN) [error_code=0000]
(XEN) Faulting linear address: 0000000000000000
(XEN) ****************************************
(XEN)
(XEN) Manual reset required (''noreboot'' specified)
------------------------------------------------------------------------
------
name = "vs1_1"
builder = "hvm"
memory = "2000"
disk = [ ''phy:/dev/vg00/vs1-1,hda,w'', ]
vif = [ ''type=ioemu, mac=00:16:3e:51:08:1a, bridge=xenbr0'',
''type=ioemu,
mac=00:16:3e:51:08:1b, bridge=xenbr1'']
uuid = "ed1618df-919f-2f47-20ea-7a5bb614e094"
device_model = "/usr/lib64/xen/bin/qemu-dm"
kernel = "/usr/lib/xen/boot/hvmloader"
vnc=1
vncunused=1
apic=1
acpi=1
pae=1
vcpus=4
serial = "pty" # enable serial console
on_reboot = ''restart''
on_crash = ''restart''
-----Original Message-----
From: Petersson, Mats [mailto:Mats.Petersson@amd.com]
Sent: Friday, May 18, 2007 12:37 PM
To: Schober Walter; xen-users@lists.xensource.com
Subject: RE: [Xen-users] How do I find out why Dom0 crashes on guest
startup?
> -----Original Message-----
> From: Schober Walter [mailto:Walter.Schober@neotel.at]
> Sent: 18 May 2007 11:35
> To: xen-users@lists.xensource.com
> Cc: Petersson, Mats
> Subject: RE: [Xen-users] How do I find out why Dom0 crashes
> on guest startup?
>
> Thanks Mats! I can do a remote serial console via iLO2 on com2: but
> cannot cut&paste from it (Hello HP!) - so I have screenshots only (can
> send them on request) ...
>
> Finally it tells me
> Panic on CPU 2:
> CPU2 FATAL PAGE FAULT
> [error_code=0000]
> Faulting linear address: 000000000000000
>
> It happens on the
> (XEN) (GUEST: 1) Starting emulated 16-bit real-mode: ip=0600:000
>
> Stack Trace can be sent on request as JPG.
I think a stack-trace would be useful here. I suspect its'' someplace in
the HVM/VMX (or SVM) code, but it''s much easier to tell what''s
going on
from a stack-trace.
--
Mats
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Nico Kadel-Garcia
2007-May-18 10:47 UTC
Re: [Xen-users] How do I find out why Dom0 crashes on guest startup?
Schober Walter wrote:> > Before Wednesday all run fine. Then I must have done something - I > tried to get rid of those IPMI error messages on starting hpasm, > installed the latest RPMs from HP website - but can''t tell exactly > when it started that the DomU startup crashed my Dom0. > > Anyway … How can I find out now, what exactly causes my Dom0 to crash? > Which logs would tell me that? > > /var/log/messages: Sometimes xenbr changes into forwarding of tap0 > before, sometimes not. > /var/log/xen/xend.log: The only thing that sounds strange: > [2007-05-18 11:56:58 xend 3475] DEBUG (XendDomain:153) number of vcpus > to use is 0 > > Meanwhile: hp-OpenIMPI, hpasm, hprsm, cmanic removed, OpenIPMI removed. > > Still, the Dom0 crashed (just reboots, no output on Display, SSH > connection interrupted) on starting a DomU. > > System: Centos 5 x86_64 @ HP DL360G5, latest yum update done. > > Will attach more info on config, when needed. > > Many thanks! > Walter >RHEL 5, and thus CentOS 5, both have auto-updates turned on by default with yum-updatesd. I consider this *insane*, since it will patch things without a chance to say "hmm, I''m busy today, let''s not do that Xen kernel patch right now!". Can you bring up the machine without Xen and check /var/log/rpmpkgs, created by /etc/cron.daily/rpm, to check the software you''ve installed? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Schober Walter
2007-May-18 11:06 UTC
RE: [Xen-users] How do I find out why Dom0 crashes on guest startup?
I have something to confess: I installed qemu (qemu-0.9.0-i386.tar.gz), too. Needed that qemu-img to convert an raw to a qcow. Removed all the files from the package now, rebooted then, but still Dom0 crashes. The guest crashing the Dom0 is not using that qcow image. br Walter -----Original Message----- From: Nico Kadel-Garcia [mailto:nkadel@gmail.com] Sent: Friday, May 18, 2007 12:47 PM To: Schober Walter Cc: xen-users@lists.xensource.com Subject: Re: [Xen-users] How do I find out why Dom0 crashes on guest startup? Schober Walter wrote:> > Before Wednesday all run fine. Then I must have done something - I > tried to get rid of those IPMI error messages on starting hpasm, > installed the latest RPMs from HP website - but can''t tell exactly > when it started that the DomU startup crashed my Dom0. > > Anyway ... How can I find out now, what exactly causes my Dom0 tocrash?> Which logs would tell me that? > > /var/log/messages: Sometimes xenbr changes into forwarding of tap0 > before, sometimes not. > /var/log/xen/xend.log: The only thing that sounds strange: > [2007-05-18 11:56:58 xend 3475] DEBUG (XendDomain:153) number of vcpus> to use is 0 > > Meanwhile: hp-OpenIMPI, hpasm, hprsm, cmanic removed, OpenIPMIremoved.> > Still, the Dom0 crashed (just reboots, no output on Display, SSH > connection interrupted) on starting a DomU. > > System: Centos 5 x86_64 @ HP DL360G5, latest yum update done. > > Will attach more info on config, when needed. > > Many thanks! > Walter >RHEL 5, and thus CentOS 5, both have auto-updates turned on by default with yum-updatesd. I consider this *insane*, since it will patch things without a chance to say "hmm, I''m busy today, let''s not do that Xen kernel patch right now!". Can you bring up the machine without Xen and check /var/log/rpmpkgs, created by /etc/cron.daily/rpm, to check the software you''ve installed? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Petersson, Mats
2007-May-18 11:21 UTC
RE: [Xen-users] How do I find out why Dom0 crashes on guest startup?
[snip big log]> (XEN) Xen call trace: > (XEN) [<ffff830000142da7>] vlapic_range+0x7/0x30So, the code here does very few things (at least in the current version, but I doubt that the older code is MUCH different). This in turn means that it''s pretty easy to determine what could be wrong: 1. VCPU = NULL - quite likely, as it''s one of the first steps. 2. vlapic field in vcpu = NULL - even more likely. I''d try running your guest with "apic=0" to see if that makes any difference. [That will of course screw up SMP on your guest, but at least it will hopefully tell us if it''s APIC that is a problem or not] I doubt that having installed a newer QEMU would make any difference. -- Mats _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Schober Walter
2007-May-18 12:10 UTC
RE: [Xen-users] How do I find out why Dom0 crashes on guest startup?
Tested: 1. apic=0, vcpus=4 OK, no crash 2. apic=1, vcpus=1 OK, no crash 3. apic=1, vcpus=2 OK (?!?!?) But: (XEN) This hvm_vlapic is for P4, no work for De-assert init (XEN) (file=hvm.c, line=665) AP 1 bringup suceeded. (XEN) vmx_do_launch(): GUEST_CR3<=00fbd7a0, HOST_CR3<=cc3ae000 (XEN) (GUEST: 4) Start AP 1 from 00006000 ... (XEN) (GUEST: 4) Starting emulated 16-bit real-mode: ip=0600:0000 4. apic=1, vcpus=4 NOK, but no crash anymore ?!?!? A lot of (XEN) <ioapic_inj_irq> error delivery mode 7 and on the guest: ..MP-BIOS bug: 8254 timer not connected to IO-APIC Kenrel panic - not syncing: IO-APIC + timer doesn''t work! Try using the noapic kernel parameter 5. apic=1, vcpus=4, ''xm create'' right after reboot NOK, Dom0 crashes as before 6. starting at least one another VM with apic=0 or commented out apic -> See 3. + 4., but no Dom0 crash. => Dom0 crashes only, if the hvm wasn''t started with apic=0 once before. That brings me back to time when I installed that VM: Installation was possible only with vcpus=1 (stale after Centos 5 Install Screen). After that I set vcpus=4 and rebootet the VM. On Wednesday I had to reboot the Dom0 for that HP management thing and started the VM the first time directly with apic=1, vcpus=4 causing the crash. So it had nothing to do with the HP drivers nor QEMU package. I guess, I have to wait for a fix in the Xen kernel or maybe try to compile my own 3.0.4. Thanks, Mats for you help! br Walter -----Original Message----- From: Petersson, Mats [mailto:Mats.Petersson@amd.com] Sent: Friday, May 18, 2007 1:22 PM To: Schober Walter; xen-users@lists.xensource.com Subject: RE: [Xen-users] How do I find out why Dom0 crashes on guest startup? [snip big log]> (XEN) Xen call trace: > (XEN) [<ffff830000142da7>] vlapic_range+0x7/0x30So, the code here does very few things (at least in the current version, but I doubt that the older code is MUCH different). This in turn means that it''s pretty easy to determine what could be wrong: 1. VCPU = NULL - quite likely, as it''s one of the first steps. 2. vlapic field in vcpu = NULL - even more likely. I''d try running your guest with "apic=0" to see if that makes any difference. [That will of course screw up SMP on your guest, but at least it will hopefully tell us if it''s APIC that is a problem or not] I doubt that having installed a newer QEMU would make any difference. -- Mats _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Petersson, Mats
2007-May-18 12:36 UTC
RE: [Xen-users] How do I find out why Dom0 crashes on guest startup?
> -----Original Message----- > From: Schober Walter [mailto:Walter.Schober@neotel.at] > Sent: 18 May 2007 13:10 > To: xen-users@lists.xensource.com > Cc: Petersson, Mats > Subject: RE: [Xen-users] How do I find out why Dom0 crashes > on guest startup? > > Tested: > 1. apic=0, vcpus=4 OK, no crash > > 2. apic=1, vcpus=1 OK, no crash > > 3. apic=1, vcpus=2 OK (?!?!?) But: > (XEN) This hvm_vlapic is for P4, no work for De-assert init > (XEN) (file=hvm.c, line=665) AP 1 bringup suceeded. > (XEN) vmx_do_launch(): GUEST_CR3<=00fbd7a0, HOST_CR3<=cc3ae000 > (XEN) (GUEST: 4) Start AP 1 from 00006000 ... > (XEN) (GUEST: 4) Starting emulated 16-bit real-mode: ip=0600:0000 > > 4. apic=1, vcpus=4 NOK, but no crash anymore ?!?!? > A lot of > (XEN) <ioapic_inj_irq> error delivery mode 7 > and on the guest: > ..MP-BIOS bug: 8254 timer not connected to IO-APIC > Kenrel panic - not syncing: IO-APIC + timer doesn''t work! Try > using the > noapic kernel parameter > > 5. apic=1, vcpus=4, ''xm create'' right after reboot NOK, Dom0 > crashes as before > > 6. starting at least one another VM with apic=0 or commented > out apic -> > See 3. + 4., but no Dom0 crash.Interesting behaviour. There was a change between 3.0.3 and 3.0.4 that changed the way that the vlapic info is stored in connection to the VCPU, and I suspect that the problem is related to a race of "adding the vlapic to the VCPU" and "using vlapic from the vcpu". Not sure why that should be, but I suspect that''s the problem.> > => Dom0 crashes only, if the hvm wasn''t started with apic=0 > once before. > > That brings me back to time when I installed that VM: Installation was > possible only with vcpus=1 (stale after Centos 5 Install > Screen). After > that I set vcpus=4 and rebootet the VM. > > On Wednesday I had to reboot the Dom0 for that HP management thing and > started the VM the first time directly with apic=1, vcpus=4 > causing the > crash. So it had nothing to do with the HP drivers nor QEMU package. > > I guess, I have to wait for a fix in the Xen kernel or maybe try to > compile my own 3.0.4.I''d try the xen-3.1-testing.hg repository - it''s closed for anything but really critical updates now, so it''s pretty much the same as final version. That''s the latest availble (aside from "unstable", which I wouldn''t recommend unless you find that 3.1 doesn''t work either). -- Mats> > Thanks, Mats for you help! > br > Walter > > -----Original Message----- > From: Petersson, Mats [mailto:Mats.Petersson@amd.com] > Sent: Friday, May 18, 2007 1:22 PM > To: Schober Walter; xen-users@lists.xensource.com > Subject: RE: [Xen-users] How do I find out why Dom0 crashes on guest > startup? > > [snip big log] > > (XEN) Xen call trace: > > (XEN) [<ffff830000142da7>] vlapic_range+0x7/0x30 > > So, the code here does very few things (at least in the > current version, > but I doubt that the older code is MUCH different). This in turn means > that it''s pretty easy to determine what could be wrong: > 1. VCPU = NULL - quite likely, as it''s one of the first steps. > 2. vlapic field in vcpu = NULL - even more likely. > > I''d try running your guest with "apic=0" to see if that makes any > difference. [That will of course screw up SMP on your guest, but at > least it will hopefully tell us if it''s APIC that is a problem or not] > > I doubt that having installed a newer QEMU would make any difference. > > -- > Mats > > > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Schober Walter
2007-May-28 17:12 UTC
RE: [Xen-users] How do I find out why Dom0 crashes on guest startup?
BTW: Reproducable on a fresh install of Centos 5 on a different host. Problem doesn''t occur anymore in 3.1. br Walter> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > Schober Walter > Sent: Friday, May 18, 2007 2:10 PM > To: xen-users@lists.xensource.com > Cc: Petersson, Mats > Subject: RE: [Xen-users] How do I find out why Dom0 crashes > on guest startup? > > Tested: > 1. apic=0, vcpus=4 OK, no crash > > 2. apic=1, vcpus=1 OK, no crash > > 3. apic=1, vcpus=2 OK (?!?!?) But: > (XEN) This hvm_vlapic is for P4, no work for De-assert init > (XEN) (file=hvm.c, line=665) AP 1 bringup suceeded. > (XEN) vmx_do_launch(): GUEST_CR3<=00fbd7a0, HOST_CR3<=cc3ae000 > (XEN) (GUEST: 4) Start AP 1 from 00006000 ... > (XEN) (GUEST: 4) Starting emulated 16-bit real-mode: ip=0600:0000 > > 4. apic=1, vcpus=4 NOK, but no crash anymore ?!?!? > A lot of > (XEN) <ioapic_inj_irq> error delivery mode 7 > and on the guest: > ..MP-BIOS bug: 8254 timer not connected to IO-APIC > Kenrel panic - not syncing: IO-APIC + timer doesn''t work! Try > using the > noapic kernel parameter > > 5. apic=1, vcpus=4, ''xm create'' right after reboot NOK, Dom0 > crashes as before > > 6. starting at least one another VM with apic=0 or commented > out apic -> > See 3. + 4., but no Dom0 crash. > > => Dom0 crashes only, if the hvm wasn''t started with apic=0 > once before. > > That brings me back to time when I installed that VM: Installation was > possible only with vcpus=1 (stale after Centos 5 Install > Screen). After > that I set vcpus=4 and rebootet the VM. > > On Wednesday I had to reboot the Dom0 for that HP management thing and > started the VM the first time directly with apic=1, vcpus=4 > causing the > crash. So it had nothing to do with the HP drivers nor QEMU package. > > I guess, I have to wait for a fix in the Xen kernel or maybe try to > compile my own 3.0.4. > > Thanks, Mats for you help! > br > Walter > > -----Original Message----- > From: Petersson, Mats [mailto:Mats.Petersson@amd.com] > Sent: Friday, May 18, 2007 1:22 PM > To: Schober Walter; xen-users@lists.xensource.com > Subject: RE: [Xen-users] How do I find out why Dom0 crashes on guest > startup? > > [snip big log] > > (XEN) Xen call trace: > > (XEN) [<ffff830000142da7>] vlapic_range+0x7/0x30 > > So, the code here does very few things (at least in the > current version, > but I doubt that the older code is MUCH different). This in turn means > that it''s pretty easy to determine what could be wrong: > 1. VCPU = NULL - quite likely, as it''s one of the first steps. > 2. vlapic field in vcpu = NULL - even more likely. > > I''d try running your guest with "apic=0" to see if that makes any > difference. [That will of course screw up SMP on your guest, but at > least it will hopefully tell us if it''s APIC that is a problem or not] > > I doubt that having installed a newer QEMU would make any difference. > > -- > Mats > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users