thr3ads.net - Xen devel - crash in nvmx_vcpu

If this information is useful, please help other people find it:
Share via:

Olaf Hering

2013-Feb-20 14:58 UTC

crash in nvmx_vcpu_destroy

while doing "while xm migrate --live domU localhost;do sleep 1;done" I
just got the crash shown below. And it can be reproduced.

The guest has 2 vcpus and 512mb, it runs pvops 3.7.9


(XEN) ----[ Xen-4.3.26579-20130219.172714  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    14
(XEN) RIP:    e008:[<ffff82c4c01dd197>] nvmx_vcpu_destroy+0xb7/0x150
(XEN) RFLAGS: 0000000000010282   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: ffff830084309000   rcx: 0000000000000060
(XEN) rdx: 0000000000000000   rsi: 0000000000000003   rdi: fffffffffffffff8
(XEN) rbp: ffff8300843096e0   rsp: ffff83036ff37e40   r8:  0000000000000001
(XEN) r9:  0000000000000001   r10: ffff83066e349800   r11: ffff82c4c01dd0e0
(XEN) r12: ffff830084309000   r13: ffff83036d371000   r14: ffff83036d371e90
(XEN) r15: ffff82c4c02cf800   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 000000008c065000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff83036ff37e40:
(XEN)    ffff830084309000 ffff830084309000 ffff830084309000 0000000000000008
(XEN)    0000000000000001 ffff82c4c01b5ce9 ffff830084309000 ffff82c4c0104e84
(XEN)    0000000000000000 ffff83036ff4a1a0 0000000000000001 ffff82c4c02cf800
(XEN)    ffff82c4c02d7800 ffff82c4c012c639 000000000000000e ffffffffffffffff
(XEN)    ffff83036ff30000 ffff82c4c0124777 ffff82c4c0124777 ffff83036ff30000
(XEN)    000000000000000e ffff82c4c02f2800 ffff83008c201000 0000000000000000
(XEN)    ffff83036ff4a060 ffff82c4c015c295 ffff83008c216000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 00007fc343205d98
(XEN)    0000000000000000 00007fc342f1764a 0000000000000000 0000000000100000
(XEN)    0000000001fb1500 00007fc33ee93010 0000000000000000 0000000000000000
(XEN)    0000000000000000 00007fc340ee1a90 000000fa00000000 00007fc342f17bc9
(XEN)    000000000000e033 0000000000000202 00007fff25f3ede8 000000000000e02b
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    000000000000000e ffff83008c216000 0000003eafc41c00 0000000000000000
(XEN) Xen call trace:
(XEN)    [<ffff82c4c01dd197>] nvmx_vcpu_destroy+0xb7/0x150
(XEN)    [<ffff82c4c01b5ce9>] hvm_vcpu_destroy+0x9/0x40
(XEN)    [<ffff82c4c0104e84>] complete_domain_destroy+0x54/0x180
(XEN)    [<ffff82c4c012c639>] rcu_process_callbacks+0xb9/0x230
(XEN)    [<ffff82c4c0124777>] __do_softirq+0x67/0xa0
(XEN)    [<ffff82c4c0124777>] __do_softirq+0x67/0xa0
(XEN)    [<ffff82c4c015c295>] idle_loop+0x25/0x50
(XEN)
(XEN) Pagetable walk from 0000000000000000:
(XEN)  L4[0x000] = 000000036fff2063 ffffffffffffffff
(XEN)  L3[0x000] = 000000036fff1063 ffffffffffffffff
(XEN)  L2[0x000] = 000000036fff0063 ffffffffffffffff
(XEN)  L1[0x000] = 0000000000000000 ffffffffffffffff
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 14:
(XEN) FATAL PAGE FAULT
(XEN) [error_code=0000]
(XEN) Faulting linear address: 0000000000000000
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...
(XEN) Resetting with ACPI MEMORY or I/O RESET_REG.



Normal dom0 bootlog is like this:

  Booting ''Xen -- SUSE Linux Enterprise Server 11 SP2 -
3.0.58-0.6.2''

root (hd0,5)
 Filesystem type is ext2fs, partition type 0x83
kernel /boot/xen.gz console=com1 com1=57600 loglvl=all guest_loglvl=all
   [Multiboot-elf, <0x100000:0x1ace98:0x5d168>, shtab=0x30a078,
entry=0x100000]
module /boot/vmlinuz-3.0.58-0.6.2-xen quiet sysrq=yes panic=9 console=ttyS0,576
00 resume=/dev/disk/by-id/ata-ST9500530NS_9SP1KKAS-part2 splash=silent showopts
 log_buf_len=64M
   [Multiboot-module @ 0x30b000, 0xb8fc80 bytes]
module /boot/initrd-3.0.58-0.6.2-xen
   [Multiboot-module @ 0xe9b000, 0x1112400 bytes]

 __  __
 \ \/ /___ _ __
  \  // _ \ ''_ \
  /  \  __/ | | |
 /_/\_\___|_| |_|

  _  _    _____  ____   __  ____ _____ ___    ____   ___  _ _____  ___ ____  _
 | || |  |___ / |___ \ / /_| ___|___  / _ \  |___ \ / _ \/ |___ / / _ \___ \/ |
 | || |_   |_ \   __) | ''_ \___ \  / / (_) |__ __) | | | | | |_ \| | |
|__) | |
 |__   _| ___) | / __/| (_) |__) |/ / \__, |__/ __/| |_| | |___) | |_| / __/| |
    |_|(_)____(_)_____|\___/____//_/    /_/  |_____|\___/|_|____/ \___/_____|_|

   ___   _ _____ ____ _____ _ _  _
  / _ \ / |___  |___ \___  / | || |
 | (_) || |  / /  __) | / /| | || |_
  \__, || | / /  / __/ / / | |__   _|
    /_(_)_|/_/  |_____/_/  |_|  |_|

(XEN) Xen version 4.3.26579-20130219.172714 (abuild@) (gcc (SUSE Linux) 4.3.4
[gcc-4_3-branch revision 152973]) debug=n Tue Feb 19 17:31:26 UTC 2013
(XEN) Latest ChangeSet: unavailable
(XEN) Bootloader: GNU GRUB 0.97
(XEN) Command line: console=com1 com1=57600 loglvl=all guest_loglvl=all
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 0 seconds
(XEN)  EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN)  Found 1 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009b000 (usable)
(XEN)  000000000009b000 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 000000008c21c000 (usable)
(XEN)  000000008c21c000 - 000000008c2ef000 (ACPI NVS)
(XEN)  000000008c2ef000 - 000000008c3df000 (ACPI data)
(XEN)  000000008c3df000 - 000000008d7df000 (ACPI NVS)
(XEN)  000000008d7df000 - 000000008f302000 (ACPI data)
(XEN)  000000008f302000 - 000000008f34f000 (reserved)
(XEN)  000000008f34f000 - 000000008f3d4000 (ACPI data)
(XEN)  000000008f3d4000 - 000000008f3de000 (ACPI NVS)
(XEN)  000000008f3de000 - 000000008f3e2000 (ACPI data)
(XEN)  000000008f3e2000 - 000000008f4cf000 (ACPI NVS)
(XEN)  000000008f4cf000 - 000000008f500000 (ACPI data)
(XEN)  000000008f500000 - 0000000090000000 (reserved)
(XEN)  00000000a0000000 - 00000000b0000000 (reserved)
(XEN)  00000000fc000000 - 00000000fd000000 (reserved)
(XEN)  00000000fed1c000 - 00000000fed20000 (reserved)
(XEN)  00000000ff800000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000670000000 (usable)
(XEN) ACPI: RSDP 000F0410, 0024 (r2 INTEL )
(XEN) ACPI: XSDT 8F4FD120, 00A4 (r1 INTEL  S5520UT         0       1000013)
(XEN) ACPI: FACP 8F4FB000, 00F4 (r4 INTEL  S5520UT         0 MSFT  100000D)
(XEN) ACPI: DSDT 8F4F4000, 65E9 (r2 INTEL  S5520UT         3 MSFT  100000D)
(XEN) ACPI: FACS 8F3E2000, 0040
(XEN) ACPI: APIC 8F4F3000, 01A8 (r2 INTEL  S5520UT         0 MSFT  100000D)
(XEN) ACPI: MCFG 8F4F2000, 003C (r1 INTEL  S5520UT         1 MSFT  100000D)
(XEN) ACPI: HPET 8F4F1000, 0038 (r1 INTEL  S5520UT         1 MSFT  100000D)
(XEN) ACPI: SLIT 8F4F0000, 0030 (r1 INTEL  S5520UT         1 MSFT  100000D)
(XEN) ACPI: SRAT 8F4EF000, 0430 (r2 INTEL  S5520UT         1 MSFT  100000D)
(XEN) ACPI: SPCR 8F4EE000, 0050 (r1 INTEL  S5520UT         0 MSFT  100000D)
(XEN) ACPI: WDDT 8F4ED000, 0040 (r1 INTEL  S5520UT         0 MSFT  100000D)
(XEN) ACPI: SSDT 8F4D2000, 1AFC4 (r2  INTEL SSDT  PM     4000 INTL 20061109)
(XEN) ACPI: SSDT 8F4D1000, 01D8 (r2  INTEL IPMI         4000 INTL 20061109)
(XEN) ACPI: TCPA 8F4D0000, 0032 (r0                        0             0)
(XEN) ACPI: HEST 8F4CF000, 00A8 (r1 INTEL  S5520UT         1 INTL        1)
(XEN) ACPI: BERT 8F3E1000, 0030 (r1 INTEL  S5520UT         1 INTL        1)
(XEN) ACPI: ERST 8F3E0000, 0230 (r1 INTEL  S5520UT         1 INTL        1)
(XEN) ACPI: EINJ 8F3DF000, 0130 (r1 INTEL  S5520UT         1 INTL        1)
(XEN) ACPI: DMAR 8F3DE000, 01A8 (r1 INTEL  S5520UT         1 MSFT  100000D)
(XEN) System RAM: 24513MB (25102044kB)
(XEN) SRAT: PXM 0 -> APIC 0 -> Node 0
(XEN) SRAT: PXM 1 -> APIC 32 -> Node 1
(XEN) SRAT: PXM 0 -> APIC 2 -> Node 0
(XEN) SRAT: PXM 1 -> APIC 34 -> Node 1
(XEN) SRAT: PXM 0 -> APIC 4 -> Node 0
(XEN) SRAT: PXM 1 -> APIC 36 -> Node 1
(XEN) SRAT: PXM 0 -> APIC 16 -> Node 0
(XEN) SRAT: PXM 1 -> APIC 48 -> Node 1
(XEN) SRAT: PXM 0 -> APIC 18 -> Node 0
(XEN) SRAT: PXM 1 -> APIC 50 -> Node 1
(XEN) SRAT: PXM 0 -> APIC 20 -> Node 0
(XEN) SRAT: PXM 1 -> APIC 52 -> Node 1
(XEN) SRAT: PXM 0 -> APIC 1 -> Node 0
(XEN) SRAT: PXM 1 -> APIC 33 -> Node 1
(XEN) SRAT: PXM 0 -> APIC 3 -> Node 0
(XEN) SRAT: PXM 1 -> APIC 35 -> Node 1
(XEN) SRAT: PXM 0 -> APIC 5 -> Node 0
(XEN) SRAT: PXM 1 -> APIC 37 -> Node 1
(XEN) SRAT: PXM 0 -> APIC 17 -> Node 0
(XEN) SRAT: PXM 1 -> APIC 49 -> Node 1
(XEN) SRAT: PXM 0 -> APIC 19 -> Node 0
(XEN) SRAT: PXM 1 -> APIC 51 -> Node 1
(XEN) SRAT: PXM 0 -> APIC 21 -> Node 0
(XEN) SRAT: PXM 1 -> APIC 53 -> Node 1
(XEN) SRAT: Node 0 PXM 0 0-90000000
(XEN) SRAT: Node 0 PXM 0 100000000-370000000
(XEN) SRAT: Node 1 PXM 1 370000000-670000000
(XEN) NUMA: Allocated memnodemap from 66e359000 - 66e35a000
(XEN) NUMA: Using 16 for the hash shift.
(XEN) Domain heap initialised DMA width 31 bits
(XEN) found SMP MP-table at 000fdb60
(XEN) DMI 2.5 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x408
(XEN) ACPI: ACPI SLEEP INFO: pm1x_cnt[404,0], pm1x_evt[400,0]
(XEN) ACPI:                  wakeup_vec[8f3e200c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) Processor #0 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x20] enabled)
(XEN) Processor #32 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
(XEN) Processor #2 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x22] enabled)
(XEN) Processor #34 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
(XEN) Processor #4 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x24] enabled)
(XEN) Processor #36 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x10] enabled)
(XEN) Processor #16 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x30] enabled)
(XEN) Processor #48 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x12] enabled)
(XEN) Processor #18 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x09] lapic_id[0x32] enabled)
(XEN) Processor #50 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x14] enabled)
(XEN) Processor #20 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x34] enabled)
(XEN) Processor #52 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x01] enabled)
(XEN) Processor #1 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x21] enabled)
(XEN) Processor #33 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x03] enabled)
(XEN) Processor #3 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x23] enabled)
(XEN) Processor #35 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x10] lapic_id[0x05] enabled)
(XEN) Processor #5 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x11] lapic_id[0x25] enabled)
(XEN) Processor #37 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x12] lapic_id[0x11] enabled)
(XEN) Processor #17 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x13] lapic_id[0x31] enabled)
(XEN) Processor #49 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x14] lapic_id[0x13] enabled)
(XEN) Processor #19 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x15] lapic_id[0x33] enabled)
(XEN) Processor #51 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x16] lapic_id[0x15] enabled)
(XEN) Processor #21 6:12 APIC version 21
(XEN) ACPI: LAPIC (acpi_id[0x17] lapic_id[0x35] enabled)
(XEN) Processor #53 6:12 APIC version 21
(XEN) ACPI: LAPIC_NMI (acpi_id[0x00] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x02] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x03] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x04] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x05] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x06] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x07] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x08] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x09] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0a] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0b] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0c] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0d] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0e] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x0f] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x10] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x11] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x12] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x13] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x14] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x15] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x16] high level lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x17] high level lint[0x1])
(XEN) Overriding APIC driver with bigsmp
(XEN) ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: IOAPIC (id[0x09] address[0xfec90000] gsi_base[24])
(XEN) IOAPIC[1]: apic_id 9, version 32, address 0xfec90000, GSI 24-47
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode:  Phys.  Using 2 I/O APICs
(XEN) ACPI: HPET id: 0x8086a401 base: 0xfed00000
(XEN) Xen ERST support is initialized.
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 24 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 48 GSI, 4576 MSI/MSI-X
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2926.411 MHz processor.
(XEN) Initing memory sharing.
(XEN) mce_intel.c:719: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0 extended
MCE MSR 0
(XEN) Intel machine check reporting enabled
(XEN) PCI: MCFG configuration 0: base a0000000 segment 0000 buses 00 - ff
(XEN) PCI: MCFG area at a0000000 reserved in E820
(XEN) PCI: Using MCFG for segment 0000 bus 00-ff
(XEN) Intel VT-d supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) Platform timer is 14.318MHz HPET
(XEN) Defaulting to alternative key handling; send ''A'' to
switch to normal mode.
(XEN) Allocated console ring of 256 KiB.
(XEN) mwait-idle: MWAIT substates: 0x1120
(XEN) mwait-idle: v0.4 model 0x2c
(XEN) mwait-idle: lapic_timer_reliable_states 0xffffffff
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Brought up 24 CPUs
(XEN) ACPI sleep modes: S3
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, lsb, paddr 0x2000 -> 0x87c000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   000000065e000000->000000065f000000 (6150689 pages to be
allocated)
(XEN)  Init. ramdisk: 000000066eeed000->000000066ffff400
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff80002000->ffffffff8087c000
(XEN)  Init. ramdisk: 0000000000000000->0000000000000000
(XEN)  Phys-Mach map: ffffea0000000000->ffffea0002efd9a0
(XEN)  Start info:    ffffffff8087c000->ffffffff8087c4b4
(XEN)  Page tables:   ffffffff8087d000->ffffffff80886000
(XEN)  Boot stack:    ffffffff80886000->ffffffff80887000
(XEN)  TOTAL:         ffffffff80000000->ffffffff80c00000
(XEN)  ENTRY ADDRESS: ffffffff80002000
(XEN) Dom0 has maximum 24 VCPUs
(XEN) Scrubbing Free RAM: .done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input -> DOM0 (type ''CTRL-a'' three times
to switch input to Xen)
(XEN) Freed 264kB init memory.
[    0.227741] pci_root PNP0A08:00: address space collision: host bridge window
[mem 0x000c4000-0x000cbfff] conflicts with Video ROM [mem 0x000c0000-0x000c7fff]
(XEN) PCI add device 0000:00:00.0
(XEN) PCI add device 0000:00:01.0
(XEN) PCI add device 0000:00:03.0
(XEN) PCI add device 0000:00:05.0
(XEN) PCI add device 0000:00:07.0
(XEN) PCI add device 0000:00:09.0
(XEN) PCI add device 0000:00:0a.0
(XEN) PCI add:00:10.1
(XEN) PCI add device 0000:00:11.0
(XEN) PCI add device 0000:00:11.1
(XEN) PCI add device 0000:00:13.0
(XEN) PCI add device 0000:00:14.0
(XEN) PCI add device 0000:00:14.1
(XEN) PCI add device 0000:00:14.2
(XEN) PCI add device 0000:00:14.3
(XEN) PCI add device 0000:00:15.0
(XEN) PCI add device 0000:00:16.0
(XEN) PCI add device 0000:00:16.1
(XEN) PCI add device 0000:00:16.2
(XEN) PCI add device 0000:00:16.3
(XEN) PCI add device 0000:00:16.4
(XEN) PCI add device 0000:00:16.5
(XEN) PCI add device 0000:00:16.6
(XEN) PCI add device 0000:00:16.7
(XEN) PCI add device 0000:00:1a.0
(XEN) PCI add device 0000:00:1a.1
(XEN) PCI add device 0000:00:1a.2
(XEN) PCI add device 0000:00:1a.7
(XEN) PCI add device 0000:00:1c.0
(XEN) PCI add device 0000:00:1c.4
(XEN) PCI add device 0000:00:1c.5
(XEN) PCI add device 0000:00:1d.0
(XEN) PCI add device 0000:00:1d.1
(XEN) PCI add device 0000:00:1d.2
(XEN) PCI add device 0000:00:1d.7
(XEN) PCI add device 0000:00:1e.0
(XEN) PCI add device 0000:00:1f.0
(XEN) PCI add device 0000:00:1f.2
(XEN) PCI add device 0000:00:1f.3
(XEN) PCI add device 0000:01:00.0
(XEN) PCI add device 0000:01:00.1
(XEN) PCI add device 0000:05:00.0
(XEN) PCI add device 0000:06:02.0
(XEN) PCI add device 0000:06:04.0
(XEN) PCI add device 0000:09:00.0
(XEN) PCI add device 0000:0a:02.0
(XEN) PCI add device 0000:0a:04.0
(XEN) PCI add device 0000:0f:00.0
(XEN) PCI add device 0000:10:00.0
(XEN) PCI add device 0000:fe:00.0
(XEN) PCI add device 0000:fe:00.1
(XEN) PCI add device 0000:fe:02.0
(XEN) PCI add device 0000:fe:02.1
(XEN) PCI add device 0000:fe:02.2
(XEN) PCI add device 0000:fe:02.3
(XEN) PCI add device 0000:fe:02.4
(XEN) PCI add device 0000:fe:02.5
(XEN) PCI add device 0000:fe:03.0
(XEN) PCI add device 0000:fe:03.1
(XEN) PCI add device 0000:fe:03.2
(XEN) PCI add device 0000:fe:03.4
(XEN) PCI add device 0000:fe:04.0
(XEN) PCI add device 0000:fe:04.1
(XEN) PCI add device 0000:fe:04.2
(XEN) PCI add device 0000:fe:04.3
(XEN) PCI add device 0000:fe:05.0
(XEN) PCI add device 0000:fe:05.1
(XEN) PCI add device 0000:fe:05.2
(XEN) PCI add device 0000:fe:05.3
(XEN) PCI add device 0000:fe:06.0
(XEN) PCI add device 0000:fe:06.1
(XEN) PCI add device 0000:fe:06.2
(XEN) PCI add device 0000:fe:06.3
(XEN) PCI add device 0000:ff:00.0
(XEN) PCI add device 0000:ff:00.1
(XEN) PCI add device 0000:ff:02.0
(XEN) PCI add device 0000:ff:02.1
(XEN) PCI add device 0000:ff:02.2
(XEN) PCI add device 0000:ff:02.3
(XEN) PCI add device 0000:ff:02.4
(XEN) PCI add device 0000:ff:02.5
(XEN) PCI add device 0000:ff:03.0
(XEN) PCI add device 0000:ff:03.1
(XEN) PCI add device 0000:ff:03.2
(XEN) PCI add device 0000:ff:03.4
(XEN) PCI add device 0000:ff:04.0
(XEN) PCI add device 0000:ff:04.1
(XEN) PCI add device 0000:ff:04.2
(XEN) PCI add device 0000:ff:04.3
(XEN) PCI add device 0000:ff:05.0
(XEN) PCI add device 0000:ff:05.1
(XEN) PCI add device 0000:ff:05.2
(XEN) PCI add device 0000:ff:05.3
(XEN) PCI add device 0000:ff:06.0
(XEN) PCI add device 0000:ff:06.1
(XEN) PCI add device 0000:ff:06.2
(XEN) PCI add device 0000:ff:06.3
[    1.411175] i8042: No controller found

Olaf Hering

2013-Feb-21 10:49 UTC

head link

Re: crash in nvmx_vcpu_destroy

On Wed, Feb 20, Olaf Hering wrote:
> (XEN) Xen call trace:
> (XEN)    [<ffff82c4c01dd197>] nvmx_vcpu_destroy+0xb7/0x150
> (XEN)    [<ffff82c4c01b5ce9>] hvm_vcpu_destroy+0x9/0x40
For some reason nestedhvm_vcpu_destroy is not in the backtrace. And its
not clear why nestedhvm_vcpu_destroy calls into nvmx_vcpu_destroy
anyway. nestedhvm is not in the config file.

Olaf

Tim Deegan

2013-Feb-21 11:01 UTC

head link

Re: crash in nvmx_vcpu_destroy

(Cc''ing the vmx maintainers)

At 15:58 +0100 on 20 Feb (1361375903), Olaf Hering
wrote:> while doing "while xm migrate --live domU localhost;do sleep
1;done" I
> just got the crash shown below. And it can be reproduced.
> 
> The guest has 2 vcpus and 512mb, it runs pvops 3.7.9
Anything interesting printed before the crash?  My best guess by code
inspection is that nvmx->launched_list never got initialized, because of
some failure in vcpu init.

Also, if you have the xen-syms for this image, can you extract a
file/line-number for the crashing %rip (ffff82c4c01dd197)?
I''d expect it to be vvmx.c:150 or thereabouts.

And thirdly, can you try the attached patch?

Cheers

Tim.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Jan Beulich

2013-Feb-21 11:03 UTC

head link

Re: crash in nvmx_vcpu_destroy

>>> On 21.02.13 at 11:49, Olaf Hering <olaf@aepfle.de> wrote:
> On Wed, Feb 20, Olaf Hering wrote:
> 
>> (XEN) Xen call trace:
>> (XEN)    [<ffff82c4c01dd197>] nvmx_vcpu_destroy+0xb7/0x150
>> (XEN)    [<ffff82c4c01b5ce9>] hvm_vcpu_destroy+0x9/0x40
> 
> For some reason nestedhvm_vcpu_destroy is not in the backtrace.
Tail call optimization?
> And its
> not clear why nestedhvm_vcpu_destroy calls into nvmx_vcpu_destroy
> anyway. nestedhvm is not in the config file.
nHVM can be turned on and off on a domain, so cleanup is needed
unconditionally. This was also the reason why 26507:4f53ddbee940
had to revert 26503:69398345c10e.

Is this perhaps dying on the uninitialized list head? If so, I''m about
to submit a patch that ought to deal with that (albeit its original
purpose is another one).

Jan

Tim Deegan

2013-Feb-21 11:08 UTC

head link

Re: crash in nvmx_vcpu_destroy

At 11:49 +0100 on 21 Feb (1361447383), Olaf Hering
wrote:> On Wed, Feb 20, Olaf Hering wrote:
> 
> > (XEN) Xen call trace:
> > (XEN)    [<ffff82c4c01dd197>] nvmx_vcpu_destroy+0xb7/0x150
> > (XEN)    [<ffff82c4c01b5ce9>] hvm_vcpu_destroy+0x9/0x40
> 
> For some reason nestedhvm_vcpu_destroy is not in the backtrace.
On non-debug builds, nestedhvm_vcpu_destroy gets compiled using a tail
call (i.e. it jumps to the arch-specific function and lets that return
directly to the caller).
> And its not clear why nestedhvm_vcpu_destroy calls into
> nvmx_vcpu_destroy anyway. nestedhvm is not in the config file.
Intriguing.  I wonder what sets the hvm-param then.

Tim.

Tim Deegan

2013-Feb-21 11:10 UTC

head link

Re: crash in nvmx_vcpu_destroy

At 11:01 +0000 on 21 Feb (1361444479), Tim Deegan wrote:> (Cc''ing the vmx maintainers)
> 
> At 15:58 +0100 on 20 Feb (1361375903), Olaf Hering wrote:
> > while doing "while xm migrate --live domU localhost;do sleep
1;done" I
> > just got the crash shown below. And it can be reproduced.
> > 
> > The guest has 2 vcpus and 512mb, it runs pvops 3.7.9
> 
> Anything interesting printed before the crash?  My best guess by code
> inspection is that nvmx->launched_list never got initialized, because of
> some failure in vcpu init.
> 
> Also, if you have the xen-syms for this image, can you extract a
> file/line-number for the crashing %rip (ffff82c4c01dd197)?
> I''d expect it to be vvmx.c:150 or thereabouts.
> 
> And thirdly, can you try the attached patch?
Oops - not sure what I tested before , but that one doesn''t even
compile!  Try this instead.

Tim.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Jan Beulich

2013-Feb-21 11:29 UTC

head link

Re: crash in nvmx_vcpu_destroy

>>> On 21.02.13 at 12:03, "Jan Beulich"
<JBeulich@suse.com> wrote:
>> On Wed, Feb 20, Olaf Hering wrote:
>> And its
>> not clear why nestedhvm_vcpu_destroy calls into nvmx_vcpu_destroy
>> anyway. nestedhvm is not in the config file.
> 
> nHVM can be turned on and off on a domain, so cleanup is needed
> unconditionally. This was also the reason why 26507:4f53ddbee940
> had to revert 26503:69398345c10e.
Sorry, I was wrong with that for what''s already committed, as I
looked at the code with the patch just sent already in place.
Something must be setting the flag for you, unless your code
base is from the small window between the commit of that patch''s
v1 and its revert.

Jan

Olaf Hering

2013-Feb-21 14:19 UTC

head link

Re: crash in nvmx_vcpu_destroy

On Thu, Feb 21, Tim Deegan wrote:
> At 11:01 +0000 on 21 Feb (1361444479), Tim Deegan wrote:
> > (Cc''ing the vmx maintainers)
> > 
> > At 15:58 +0100 on 20 Feb (1361375903), Olaf Hering wrote:
> > > while doing "while xm migrate --live domU localhost;do sleep
1;done" I
> > > just got the crash shown below. And it can be reproduced.
> > > 
> > > The guest has 2 vcpus and 512mb, it runs pvops 3.7.9
> > 
> > Anything interesting printed before the crash?  My best guess by code
> > inspection is that nvmx->launched_list never got initialized,
because of
> > some failure in vcpu init.
> > 
> > Also, if you have the xen-syms for this image, can you extract a
> > file/line-number for the crashing %rip (ffff82c4c01dd197)?
> > I''d expect it to be vvmx.c:150 or thereabouts.
> > 
> > And thirdly, can you try the attached patch?
> 
> Oops - not sure what I tested before , but that one doesn''t even
> compile!  Try this instead.
This patch fixes the crash for me. Thanks.

Olaf
> diff --git a/xen/arch/x86/hvm/vmx/vvmx.c b/xen/arch/x86/hvm/vmx/vvmx.c
> index 4f3f94d..5d00ff7 100644
> --- a/xen/arch/x86/hvm/vmx/vvmx.c
> +++ b/xen/arch/x86/hvm/vmx/vvmx.c
> @@ -147,10 +147,13 @@ void nvmx_vcpu_destroy(struct vcpu *v)
>          nvcpu->nv_n2vmcx = NULL;
>      }
>  
> -    list_for_each_entry_safe(item, n, &nvmx->launched_list, node)
> +    if ( nvmx->launched_list.next )
>      {
> -        list_del(&item->node);
> -        xfree(item);
> +        list_for_each_entry_safe(item, n, &nvmx->launched_list,
node)
> +        {
> +            list_del(&item->node);
> +            xfree(item);
> +        }
>      }
>  
>      if ( v->arch.hvm_vmx.vmread_bitmap )

Tim Deegan

2013-Feb-21 14:22 UTC

head link

Re: crash in nvmx_vcpu_destroy

At 15:19 +0100 on 21 Feb (1361459959), Olaf Hering
wrote:> This patch fixes the crash for me. Thanks.
Great - in that case it will be fixed by Jan''s more comprehensive patch
when that goes in.

Cheers,

Tim.

Seemingly Similar Threads

Search for more reasonably related threads

Xen devel - Feb 2013 - crash in nvmx_vcpu_destroy

crash in nvmx_vcpu_destroy

Re: crash in nvmx_vcpu_destroy

Re: crash in nvmx_vcpu_destroy

Re: crash in nvmx_vcpu_destroy

Re: crash in nvmx_vcpu_destroy

Re: crash in nvmx_vcpu_destroy

Re: crash in nvmx_vcpu_destroy

Re: crash in nvmx_vcpu_destroy

Re: crash in nvmx_vcpu_destroy

Seemingly Similar Threads