Yufang Zhang
2012-Jan-11 08:03 UTC
Cannot start up HVM guest when maxmem is not equal to memory and HAP is enabled.
Hi all, On my production deployment with xen-3.4.4-rc1, HVM guest would not start up when maxmem is set not equal to memory, unless HAP is disable(hap=0). The guest just hangs at ''Booting from Hard Disk...'' as shown in the attachment. I find that both Oracle VM and Fedora have the same problem per the following links: http://docs.oracle.com/cd/E15458_01/doc.22/e15443/toc.htm#BABBIBFD http://docs.oracle.com/cd/E26996_01/e18546/CJAGDAGB.html#id419003 https://bugzilla.redhat.com/show_bug.cgi?id=645319 Could I ask if there is any work in upstream/xen-unstable to fix this problem? Thanks. Below is the xm dmesg output related with the hvm guest: (XEN) HVM3: HVM Loader (XEN) HVM3: Detected Xen v3.4.4-rc1-pre (XEN) HVM3: CPU speed is 2267 MHz (XEN) irq.c:243: Dom3 PCI link 0 changed 0 -> 5 (XEN) HVM3: PCI-ISA link 0 routed to IRQ5 (XEN) irq.c:243: Dom3 PCI link 1 changed 0 -> 10 (XEN) HVM3: PCI-ISA link 1 routed to IRQ10 (XEN) irq.c:243: Dom3 PCI link 2 changed 0 -> 11 (XEN) HVM3: PCI-ISA link 2 routed to IRQ11 (XEN) irq.c:243: Dom3 PCI link 3 changed 0 -> 5 (XEN) HVM3: PCI-ISA link 3 routed to IRQ5 (XEN) HVM3: pci dev 01:2 INTD->IRQ5 (XEN) HVM3: pci dev 01:3 INTA->IRQ10 (XEN) HVM3: pci dev 03:0 INTA->IRQ5 (XEN) HVM3: pci dev 04:0 INTA->IRQ5 (XEN) HVM3: pci dev 02:0 bar 10 size 02000000: f0000008 (XEN) HVM3: pci dev 03:0 bar 14 size 01000000: f2000008 (XEN) HVM3: pci dev 02:0 bar 14 size 00001000: f3000000 (XEN) HVM3: pci dev 03:0 bar 10 size 00000100: 0000c001 (XEN) HVM3: pci dev 04:0 bar 10 size 00000100: 0000c101 (XEN) HVM3: pci dev 04:0 bar 14 size 00000100: f3001000 (XEN) HVM3: pci dev 01:2 bar 20 size 00000020: 0000c201 (XEN) HVM3: pci dev 01:1 bar 20 size 00000010: 0000c221 (XEN) HVM3: Multiprocessor initialisation: (XEN) HVM3: - CPU0 ... 40-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done. (XEN) HVM3: Writing SMBIOS tables ... (XEN) HVM3: Loading ROMBIOS ... (XEN) HVM3: 10332 bytes of ROMBIOS high-memory extensions: (XEN) HVM3: Relocating to 0xfc000000-0xfc00285c ... done (XEN) HVM3: Creating MP tables ... (XEN) HVM3: Loading Cirrus VGABIOS ... (XEN) HVM3: Loading PCI Option ROM ... (XEN) HVM3: - Manufacturer: http://etherboot.org (XEN) HVM3: - Product name: gPXE (XEN) HVM3: Loading ACPI ... (XEN) HVM3: - Lo data: 000ea020-000ea04f (XEN) HVM3: - Hi data: fc002c00-fc0060bf (XEN) HVM3: vm86 TSS at fc006400 (XEN) HVM3: BIOS map: (XEN) HVM3: c0000-c8fff: VGA BIOS (XEN) HVM3: c9000-d57ff: Etherboot ROM (XEN) HVM3: eb000-eb150: SMBIOS tables (XEN) HVM3: f0000-fffff: Main BIOS (XEN) HVM3: Invoking ROMBIOS ... (XEN) HVM3: $Revision: 1.221 $ $Date: 2008/12/07 17:32:29 $ (XEN) stdvga.c:147:d3 entering stdvga and caching modes (XEN) HVM3: VGABios $Id: vgabios.c,v 1.67 2008/01/27 09:44:12 vruppert Exp $ (XEN) HVM3: Bochs BIOS - build: 06/23/99 (XEN) HVM3: $Revision: 1.221 $ $Date: 2008/12/07 17:32:29 $ (XEN) HVM3: Options: apmbios pcibios eltorito PMM (XEN) HVM3: (XEN) HVM3: ata0-0: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 (XEN) HVM3: ata0 master: QEMU HARDDISK ATA-7 Hard-Disk (8192 MBytes) (XEN) HVM3: IDE time out (XEN) HVM3: (XEN) HVM3: (XEN) HVM3: (XEN) HVM3: Press F12 for boot menu. (XEN) HVM3: (XEN) HVM3: Booting from Hard Disk... (XEN) HVM3: Booting from 0000:7c00 (XEN) io.c:199:d3 MMIO emulation failed @ 0008:4013b1: 90 86 95 21 08 83 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
George Dunlap
2012-Jan-11 11:27 UTC
Re: Cannot start up HVM guest when maxmem is not equal to memory and HAP is enabled.
On Wed, Jan 11, 2012 at 8:03 AM, Yufang Zhang <yufang521247@gmail.com> wrote:> Hi all, > > On my production deployment with xen-3.4.4-rc1, HVM guest would not start up > when maxmem is set not equal to memory, unless HAP is disable(hap=0). The > guest just hangs at ''Booting from Hard Disk...'' as shown in the attachment.Is there a reason you want to set maxmem != memory? Starting in 4.0, this will cause the VM to boot in "populate-on-demand" mode; but I think in 3.4, this may have undefined behavior.> I find that both Oracle VM and Fedora have the same problem per the > following links: > > http://docs.oracle.com/cd/E15458_01/doc.22/e15443/toc.htm#BABBIBFD > http://docs.oracle.com/cd/E26996_01/e18546/CJAGDAGB.html#id419003 > https://bugzilla.redhat.com/show_bug.cgi?id=645319 > > Could I ask if there is any work in upstream/xen-unstable to fix this > problem?Setting maxmem != memory in 4.1 will definitely have a different effect -- if your guest has a balloon driver, you will start your VM "pre-ballooned"; if your guest does not, it will crash as soon as it dirties more than "memory" amount of memory. -George> > Thanks. > > Below is the xm dmesg output related with the hvm guest: > (XEN) HVM3: HVM Loader > (XEN) HVM3: Detected Xen v3.4.4-rc1-pre > (XEN) HVM3: CPU speed is 2267 MHz > (XEN) irq.c:243: Dom3 PCI link 0 changed 0 -> 5 > (XEN) HVM3: PCI-ISA link 0 routed to IRQ5 > (XEN) irq.c:243: Dom3 PCI link 1 changed 0 -> 10 > (XEN) HVM3: PCI-ISA link 1 routed to IRQ10 > (XEN) irq.c:243: Dom3 PCI link 2 changed 0 -> 11 > (XEN) HVM3: PCI-ISA link 2 routed to IRQ11 > (XEN) irq.c:243: Dom3 PCI link 3 changed 0 -> 5 > (XEN) HVM3: PCI-ISA link 3 routed to IRQ5 > (XEN) HVM3: pci dev 01:2 INTD->IRQ5 > (XEN) HVM3: pci dev 01:3 INTA->IRQ10 > (XEN) HVM3: pci dev 03:0 INTA->IRQ5 > (XEN) HVM3: pci dev 04:0 INTA->IRQ5 > (XEN) HVM3: pci dev 02:0 bar 10 size 02000000: f0000008 > (XEN) HVM3: pci dev 03:0 bar 14 size 01000000: f2000008 > (XEN) HVM3: pci dev 02:0 bar 14 size 00001000: f3000000 > (XEN) HVM3: pci dev 03:0 bar 10 size 00000100: 0000c001 > (XEN) HVM3: pci dev 04:0 bar 10 size 00000100: 0000c101 > (XEN) HVM3: pci dev 04:0 bar 14 size 00000100: f3001000 > (XEN) HVM3: pci dev 01:2 bar 20 size 00000020: 0000c201 > (XEN) HVM3: pci dev 01:1 bar 20 size 00000010: 0000c221 > (XEN) HVM3: Multiprocessor initialisation: > (XEN) HVM3: - CPU0 ... 40-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... > done. > (XEN) HVM3: Writing SMBIOS tables ... > (XEN) HVM3: Loading ROMBIOS ... > (XEN) HVM3: 10332 bytes of ROMBIOS high-memory extensions: > (XEN) HVM3: Relocating to 0xfc000000-0xfc00285c ... done > (XEN) HVM3: Creating MP tables ... > (XEN) HVM3: Loading Cirrus VGABIOS ... > (XEN) HVM3: Loading PCI Option ROM ... > (XEN) HVM3: - Manufacturer: http://etherboot.org > (XEN) HVM3: - Product name: gPXE > (XEN) HVM3: Loading ACPI ... > (XEN) HVM3: - Lo data: 000ea020-000ea04f > (XEN) HVM3: - Hi data: fc002c00-fc0060bf > (XEN) HVM3: vm86 TSS at fc006400 > (XEN) HVM3: BIOS map: > (XEN) HVM3: c0000-c8fff: VGA BIOS > (XEN) HVM3: c9000-d57ff: Etherboot ROM > (XEN) HVM3: eb000-eb150: SMBIOS tables > (XEN) HVM3: f0000-fffff: Main BIOS > (XEN) HVM3: Invoking ROMBIOS ... > (XEN) HVM3: $Revision: 1.221 $ $Date: 2008/12/07 17:32:29 $ > (XEN) stdvga.c:147:d3 entering stdvga and caching modes > (XEN) HVM3: VGABios $Id: vgabios.c,v 1.67 2008/01/27 09:44:12 vruppert Exp $ > (XEN) HVM3: Bochs BIOS - build: 06/23/99 > (XEN) HVM3: $Revision: 1.221 $ $Date: 2008/12/07 17:32:29 $ > (XEN) HVM3: Options: apmbios pcibios eltorito PMM > (XEN) HVM3: > (XEN) HVM3: ata0-0: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 > (XEN) HVM3: ata0 master: QEMU HARDDISK ATA-7 Hard-Disk (8192 MBytes) > (XEN) HVM3: IDE time out > (XEN) HVM3: > (XEN) HVM3: > (XEN) HVM3: > (XEN) HVM3: Press F12 for boot menu. > (XEN) HVM3: > (XEN) HVM3: Booting from Hard Disk... > (XEN) HVM3: Booting from 0000:7c00 > (XEN) io.c:199:d3 MMIO emulation failed @ 0008:4013b1: 90 86 95 21 08 83 > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >
Yufang Zhang
2012-Jan-12 09:15 UTC
Re: Cannot start up HVM guest when maxmem is not equal to memory and HAP is enabled.
2012/1/11 George Dunlap <George.Dunlap@eu.citrix.com>> On Wed, Jan 11, 2012 at 8:03 AM, Yufang Zhang <yufang521247@gmail.com> > wrote: > > Hi all, > > > > On my production deployment with xen-3.4.4-rc1, HVM guest would not > start up > > when maxmem is set not equal to memory, unless HAP is disable(hap=0). The > > guest just hangs at ''Booting from Hard Disk...'' as shown in the > attachment. > > Is there a reason you want to set maxmem != memory? Starting in 4.0, > this will cause the VM to boot in "populate-on-demand" mode; but I > think in 3.4, this may have undefined behavior. > >Actually, you would be exposed to this problem whenever you restart xend and then reboot the hvm guest, even you set maxmem == memory in the config file of that guest. When xend is restarted, it tries to recreate guest information. memory_dynamic_max and memory_static_max, which are used to calculate memory and maxmem, are re-calculated from mem_kb and maxmem_kb respectively. mem_kb is not equal to maxmem_kb. Thus memory != maxmem after xend restarts, and rebooting this hvm guest later would leads guest crash/hang.> I find that both Oracle VM and Fedora have the same problem per the > > following links: > > > > http://docs.oracle.com/cd/E15458_01/doc.22/e15443/toc.htm#BABBIBFD > > http://docs.oracle.com/cd/E26996_01/e18546/CJAGDAGB.html#id419003 > > https://bugzilla.redhat.com/show_bug.cgi?id=645319 > > > > Could I ask if there is any work in upstream/xen-unstable to fix this > > problem? > > Setting maxmem != memory in 4.1 will definitely have a different > effect -- if your guest has a balloon driver, you will start your VM > "pre-ballooned"; if your guest does not, it will crash as soon as it > dirties more than "memory" amount of memory. > > -GeorgeI check the xend code of xen-unstable, the related logic didn''t change since xen-3.4. So I *guess* a hvm guest without balloon driver would crash after some time(when it dirties enough memory) after you restarting xend and rebooting the guest on xen-4.1.> > > Thanks. > > > > Below is the xm dmesg output related with the hvm guest: > > (XEN) HVM3: HVM Loader > > (XEN) HVM3: Detected Xen v3.4.4-rc1-pre > > (XEN) HVM3: CPU speed is 2267 MHz > > (XEN) irq.c:243: Dom3 PCI link 0 changed 0 -> 5 > > (XEN) HVM3: PCI-ISA link 0 routed to IRQ5 > > (XEN) irq.c:243: Dom3 PCI link 1 changed 0 -> 10 > > (XEN) HVM3: PCI-ISA link 1 routed to IRQ10 > > (XEN) irq.c:243: Dom3 PCI link 2 changed 0 -> 11 > > (XEN) HVM3: PCI-ISA link 2 routed to IRQ11 > > (XEN) irq.c:243: Dom3 PCI link 3 changed 0 -> 5 > > (XEN) HVM3: PCI-ISA link 3 routed to IRQ5 > > (XEN) HVM3: pci dev 01:2 INTD->IRQ5 > > (XEN) HVM3: pci dev 01:3 INTA->IRQ10 > > (XEN) HVM3: pci dev 03:0 INTA->IRQ5 > > (XEN) HVM3: pci dev 04:0 INTA->IRQ5 > > (XEN) HVM3: pci dev 02:0 bar 10 size 02000000: f0000008 > > (XEN) HVM3: pci dev 03:0 bar 14 size 01000000: f2000008 > > (XEN) HVM3: pci dev 02:0 bar 14 size 00001000: f3000000 > > (XEN) HVM3: pci dev 03:0 bar 10 size 00000100: 0000c001 > > (XEN) HVM3: pci dev 04:0 bar 10 size 00000100: 0000c101 > > (XEN) HVM3: pci dev 04:0 bar 14 size 00000100: f3001000 > > (XEN) HVM3: pci dev 01:2 bar 20 size 00000020: 0000c201 > > (XEN) HVM3: pci dev 01:1 bar 20 size 00000010: 0000c221 > > (XEN) HVM3: Multiprocessor initialisation: > > (XEN) HVM3: - CPU0 ... 40-bit phys ... fixed MTRRs ... var MTRRs [2/8] > ... > > done. > > (XEN) HVM3: Writing SMBIOS tables ... > > (XEN) HVM3: Loading ROMBIOS ... > > (XEN) HVM3: 10332 bytes of ROMBIOS high-memory extensions: > > (XEN) HVM3: Relocating to 0xfc000000-0xfc00285c ... done > > (XEN) HVM3: Creating MP tables ... > > (XEN) HVM3: Loading Cirrus VGABIOS ... > > (XEN) HVM3: Loading PCI Option ROM ... > > (XEN) HVM3: - Manufacturer: http://etherboot.org > > (XEN) HVM3: - Product name: gPXE > > (XEN) HVM3: Loading ACPI ... > > (XEN) HVM3: - Lo data: 000ea020-000ea04f > > (XEN) HVM3: - Hi data: fc002c00-fc0060bf > > (XEN) HVM3: vm86 TSS at fc006400 > > (XEN) HVM3: BIOS map: > > (XEN) HVM3: c0000-c8fff: VGA BIOS > > (XEN) HVM3: c9000-d57ff: Etherboot ROM > > (XEN) HVM3: eb000-eb150: SMBIOS tables > > (XEN) HVM3: f0000-fffff: Main BIOS > > (XEN) HVM3: Invoking ROMBIOS ... > > (XEN) HVM3: $Revision: 1.221 $ $Date: 2008/12/07 17:32:29 $ > > (XEN) stdvga.c:147:d3 entering stdvga and caching modes > > (XEN) HVM3: VGABios $Id: vgabios.c,v 1.67 2008/01/27 09:44:12 vruppert > Exp $ > > (XEN) HVM3: Bochs BIOS - build: 06/23/99 > > (XEN) HVM3: $Revision: 1.221 $ $Date: 2008/12/07 17:32:29 $ > > (XEN) HVM3: Options: apmbios pcibios eltorito PMM > > (XEN) HVM3: > > (XEN) HVM3: ata0-0: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 > > (XEN) HVM3: ata0 master: QEMU HARDDISK ATA-7 Hard-Disk (8192 MBytes) > > (XEN) HVM3: IDE time out > > (XEN) HVM3: > > (XEN) HVM3: > > (XEN) HVM3: > > (XEN) HVM3: Press F12 for boot menu. > > (XEN) HVM3: > > (XEN) HVM3: Booting from Hard Disk... > > (XEN) HVM3: Booting from 0000:7c00 > > (XEN) io.c:199:d3 MMIO emulation failed @ 0008:4013b1: 90 86 95 21 08 83 > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
George Dunlap
2012-Jan-13 17:19 UTC
Re: Cannot start up HVM guest when maxmem is not equal to memory and HAP is enabled.
On Thu, Jan 12, 2012 at 9:15 AM, Yufang Zhang <yufang521247@gmail.com> wrote:> Actually, you would be exposed to this problem whenever you restart xend and > then reboot the hvm guest, even you set maxmem == memory in the config file > of that guest. When xend is restarted, it tries to recreate guest > information. memory_dynamic_max and memory_static_max, which are used > to calculate memory and maxmem, are re-calculated from mem_kb and maxmem_kb > respectively. mem_kb is not equal to maxmem_kb. Thus memory != maxmem after > xend restarts, and rebooting this hvm guest later would leads guest > crash/hang.Ah, interesting. Yeah, that is certainly a bug -- not sure whom to contact about fixing that. Keith?> I check the xend code of xen-unstable, the related logic didn''t change > since xen-3.4. So I *guess* a hvm guest without balloon driver would crash > after some time(when it dirties enough memory) after you restarting xend and > rebooting the guest on xen-4.1.Should a user be changing the "memory" setting without having a balloon driver active? If the balloon driver *is* active, we''d want that setting to be preserved across reboot. Does xend know if there is an active balloon driver? If not, I think we just have to leave it as it is. -George