Bruce Edge
2009-Oct-11 14:22 UTC
[Xen-devel] xen/master pvops kernel boot consistently hangs on i7 with xen 3.4/3.5 where 2.6.30.3 git kernel works.
I''m running out of things to try. I would really appreciate a clue as to what to look at next. It''s not a full lockup as the CTRL-A*3 xen debug console still works, but the xen/master dom0 shows no signs of life. The current 2.6.31.1 pvops xen/master kernel hangs when running under Xen on a proliant the a supermicro, both X58 i7 boxes. I''ve tried booting with these xen versions (both of which work with a 2.6.30.3 dom0) http://xenbits.xensource.com/xen-unstable.hg http://xenbits.xensource.com/xen-3.4-testing.hg And with the 2.6.31.1 source from both (is there a difference?) git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git xen/master I''ve attached my .config, but here are the xen related settings: CONFIG_XEN=y CONFIG_XEN_TIME_VSYSCALL=y CONFIG_XEN_MAX_DOMAIN_MEMORY=32 CONFIG_XEN_SAVE_RESTORE=y CONFIG_XEN_DEBUG_FS=y CONFIG_XEN_DOM0=y CONFIG_XEN_PRIVILEGED_GUEST=y CONFIG_XEN_DOM0_PCI=y CONFIG_MICROCODE_XEN=y CONFIG_PCI_XEN=y CONFIG_XEN_BLKDEV_FRONTEND=y CONFIG_XEN_NETDEV_FRONTEND=y CONFIG_XEN_KBDDEV_FRONTEND=m CONFIG_HVC_XEN=y CONFIG_XEN_FBDEV_FRONTEND=m CONFIG_XEN_BALLOON=y CONFIG_XEN_SCRUB_PAGES=y CONFIG_XEN_DEV_EVTCHN=y CONFIG_XEN_BACKEND=y CONFIG_XEN_BLKDEV_BACKEND=y CONFIG_XEN_NETDEV_BACKEND=y CONFIG_XENFS=y CONFIG_XEN_COMPAT_XENFS=y CONFIG_XEN_SYS_HYPERVISOR=y CONFIG_XEN_XENBUS_FRONTEND=y CONFIG_XEN_S3=y CONFIG_XEN_GNTDEV=y I''m building the kernels with make deb-pkg, installing the resultant 2 packages, firmware and kernel, and building the initrd. I assume that that''s no different from running make; make modules_install, etc... on the target box? The boot options I''m using are: title Xen 3.5-unstable / Ubuntu 9.04, kernel 2.6.31.1 clean uuid 915b8d5f-dc27-4234-adac-26382dfef7c2 kernel /xen-3.5-unstable.gz com1=115200,8n1 console=com1 module /vmlinuz-2.6.31.1 root=UUID=e0a8abaf-ebb4-4a46-a968-36764653b87f ro console=xvc0 module /initrd.img-2.6.31.1 as well as: title Xen 3.5-unstable / Ubuntu 9.04, kernel 2.6.31.1 uuid 915b8d5f-dc27-4234-adac-26382dfef7c2 kernel /xen-3.5-unstable.gz com1=115200,8n1 console=com1 acpi=force apic=on dom0_max_vcpus=1 dom0_vcpus_pin=true iommu=1,no-intremap,passthrough loglvl=all loglvl_guest=all module /vmlinuz-2.6.31.1 root=UUID=e0a8abaf-ebb4-4a46-a968-36764653b87f ro console=xvc0 pciback.hide=(07:00.0)(07:00.1)(07:00.2)(07:00.3)(14:00.0)(14:00.1)(17:00.0)(17:00.1) acpi=force module /initrd.img-2.6.31.1 and everything in between. I''ve tried booting the xen/master 2.6.31.1 kernel on bare metal instead of under xen and there''s zero console output. Should this work, or does this kernel _only_ run under Xen? I can boot both xen versions, 3.5-unstable and 3.4-testing with a 2.6.30.3 dom0. This _does_ work, but it uses the older kernel: title Xen 3.5-unstable / Ubuntu 9.04, kernel 2.6.30.3 uuid 915b8d5f-dc27-4234-adac-26382dfef7c2 kernel /xen-3.5-unstable.gz com1=115200,8n1 console=com1 acpi=force apic=on dom0_max_vcpus=1 dom0_vcpus_pin=true iommu=1,no-intremap,passthrough loglvl=all loglvl_guest=all module /vmlinuz-2.6.30.3 root=UUID=e0a8abaf-ebb4-4a46-a968-36764653b87f ro console=xvc0 pciback.hide=(07:00.0)(07:00.1)(07:00.2)(07:00.3)(14:00.0)(14:00.1)(17:00.0)(17:00.1) acpi=force module /initrd.img-2.6.30.3 I''ve tried toggling these settings in the BIOS with no effect: - hyperthreading - exec memory protection - disabling all unneeded PCI cards - APIC Any other BIOS items I should tweak to test? This is the tail of the console output before the hang (XEN) PCI add device 17:00.0 (XEN) PCI add device 17:00.1 (XEN) PCI add device 07:00.0 (XEN) PCI add device 07:00.1 (XEN) PCI add device 07:00.2 (XEN) PCI add device 07:00.3 (XEN) PCI add device 02:00.0 (XEN) PCI add device 02:00.1 (XEN) PCI add device 03:00.0 (XEN) PCI add device 03:00.1 (XEN) PCI add device 01:03.0 (XEN) PCI add device 01:04.0 (XEN) PCI add device 01:04.2 (XEN) PCI add device 01:04.4 (XEN) PCI add device 01:04.6 (XEN) io_apic.c:2208: (XEN) ioapic_guest_write: apic=0, pin=20, irq=20 (XEN) ioapic_guest_write: new_entry=0001a014 (XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ! (XEN) io_apic.c:2208: (XEN) ioapic_guest_write: apic=0, pin=23, irq=23 (XEN) ioapic_guest_write: new_entry=0001a017 (XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ! (XEN) io_apic.c:2208: (XEN) ioapic_guest_write: apic=0, pin=22, irq=22 (XEN) ioapic_guest_write: new_entry=0001a016 (XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ! (XEN) *** Serial input -> Xen (type ''CTRL-a'' three times to switch input to DOM0) The CTRL-a does still work. What output from that can I post to help debug this? I''ve already posted the full boot output so I''ll resist the temptation to include it again. -Bruce _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yu, Ke
2009-Oct-12 06:10 UTC
RE: [Xen-devel] xen/master pvops kernel boot consistently hangs on i7 with xen 3.4/3.5 where 2.6.30.3 git kernel works.
Could you try to add " hpet=disable" option to dom0 command line. If it works, then we meet the same issue. this issue is caused by the dom0 misusing of HPET, we are cooking a patch to fix this issue. Best Regards Ke -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Bruce Edge Sent: Sunday, October 11, 2009 10:23 PM To: xen-devel@lists.xensource.com Subject: [Xen-devel] xen/master pvops kernel boot consistently hangs on i7 with xen 3.4/3.5 where 2.6.30.3 git kernel works. I''m running out of things to try. I would really appreciate a clue as to what to look at next. It''s not a full lockup as the CTRL-A*3 xen debug console still works, but the xen/master dom0 shows no signs of life. The current 2.6.31.1 pvops xen/master kernel hangs when running under Xen on a proliant the a supermicro, both X58 i7 boxes. I''ve tried booting with these xen versions (both of which work with a 2.6.30.3 dom0) http://xenbits.xensource.com/xen-unstable.hg http://xenbits.xensource.com/xen-3.4-testing.hg And with the 2.6.31.1 source from both (is there a difference?) git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git xen/master I''ve attached my .config, but here are the xen related settings: CONFIG_XEN=y CONFIG_XEN_TIME_VSYSCALL=y CONFIG_XEN_MAX_DOMAIN_MEMORY=32 CONFIG_XEN_SAVE_RESTORE=y CONFIG_XEN_DEBUG_FS=y CONFIG_XEN_DOM0=y CONFIG_XEN_PRIVILEGED_GUEST=y CONFIG_XEN_DOM0_PCI=y CONFIG_MICROCODE_XEN=y CONFIG_PCI_XEN=y CONFIG_XEN_BLKDEV_FRONTEND=y CONFIG_XEN_NETDEV_FRONTEND=y CONFIG_XEN_KBDDEV_FRONTEND=m CONFIG_HVC_XEN=y CONFIG_XEN_FBDEV_FRONTEND=m CONFIG_XEN_BALLOON=y CONFIG_XEN_SCRUB_PAGES=y CONFIG_XEN_DEV_EVTCHN=y CONFIG_XEN_BACKEND=y CONFIG_XEN_BLKDEV_BACKEND=y CONFIG_XEN_NETDEV_BACKEND=y CONFIG_XENFS=y CONFIG_XEN_COMPAT_XENFS=y CONFIG_XEN_SYS_HYPERVISOR=y CONFIG_XEN_XENBUS_FRONTEND=y CONFIG_XEN_S3=y CONFIG_XEN_GNTDEV=y I''m building the kernels with make deb-pkg, installing the resultant 2 packages, firmware and kernel, and building the initrd. I assume that that''s no different from running make; make modules_install, etc... on the target box? The boot options I''m using are: title Xen 3.5-unstable / Ubuntu 9.04, kernel 2.6.31.1 clean uuid 915b8d5f-dc27-4234-adac-26382dfef7c2 kernel /xen-3.5-unstable.gz com1=115200,8n1 console=com1 module /vmlinuz-2.6.31.1 root=UUID=e0a8abaf-ebb4-4a46-a968-36764653b87f ro console=xvc0 module /initrd.img-2.6.31.1 as well as: title Xen 3.5-unstable / Ubuntu 9.04, kernel 2.6.31.1 uuid 915b8d5f-dc27-4234-adac-26382dfef7c2 kernel /xen-3.5-unstable.gz com1=115200,8n1 console=com1 acpi=force apic=on dom0_max_vcpus=1 dom0_vcpus_pin=true iommu=1,no-intremap,passthrough loglvl=all loglvl_guest=all module /vmlinuz-2.6.31.1 root=UUID=e0a8abaf-ebb4-4a46-a968-36764653b87f ro console=xvc0 pciback.hide=(07:00.0)(07:00.1)(07:00.2)(07:00.3)(14:00.0)(14:00.1)(17:00.0)(17:00.1) acpi=force module /initrd.img-2.6.31.1 and everything in between. I''ve tried booting the xen/master 2.6.31.1 kernel on bare metal instead of under xen and there''s zero console output. Should this work, or does this kernel _only_ run under Xen? I can boot both xen versions, 3.5-unstable and 3.4-testing with a 2.6.30.3 dom0. This _does_ work, but it uses the older kernel: title Xen 3.5-unstable / Ubuntu 9.04, kernel 2.6.30.3 uuid 915b8d5f-dc27-4234-adac-26382dfef7c2 kernel /xen-3.5-unstable.gz com1=115200,8n1 console=com1 acpi=force apic=on dom0_max_vcpus=1 dom0_vcpus_pin=true iommu=1,no-intremap,passthrough loglvl=all loglvl_guest=all module /vmlinuz-2.6.30.3 root=UUID=e0a8abaf-ebb4-4a46-a968-36764653b87f ro console=xvc0 pciback.hide=(07:00.0)(07:00.1)(07:00.2)(07:00.3)(14:00.0)(14:00.1)(17:00.0)(17:00.1) acpi=force module /initrd.img-2.6.30.3 I''ve tried toggling these settings in the BIOS with no effect: - hyperthreading - exec memory protection - disabling all unneeded PCI cards - APIC Any other BIOS items I should tweak to test? This is the tail of the console output before the hang (XEN) PCI add device 17:00.0 (XEN) PCI add device 17:00.1 (XEN) PCI add device 07:00.0 (XEN) PCI add device 07:00.1 (XEN) PCI add device 07:00.2 (XEN) PCI add device 07:00.3 (XEN) PCI add device 02:00.0 (XEN) PCI add device 02:00.1 (XEN) PCI add device 03:00.0 (XEN) PCI add device 03:00.1 (XEN) PCI add device 01:03.0 (XEN) PCI add device 01:04.0 (XEN) PCI add device 01:04.2 (XEN) PCI add device 01:04.4 (XEN) PCI add device 01:04.6 (XEN) io_apic.c:2208: (XEN) ioapic_guest_write: apic=0, pin=20, irq=20 (XEN) ioapic_guest_write: new_entry=0001a014 (XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ! (XEN) io_apic.c:2208: (XEN) ioapic_guest_write: apic=0, pin=23, irq=23 (XEN) ioapic_guest_write: new_entry=0001a017 (XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ! (XEN) io_apic.c:2208: (XEN) ioapic_guest_write: apic=0, pin=22, irq=22 (XEN) ioapic_guest_write: new_entry=0001a016 (XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ! (XEN) *** Serial input -> Xen (type ''CTRL-a'' three times to switch input to DOM0) The CTRL-a does still work. What output from that can I post to help debug this? I''ve already posted the full boot output so I''ll resist the temptation to include it again. -Bruce _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Bruce Edge
2009-Oct-12 13:53 UTC
Re: [Xen-devel] xen/master pvops kernel boot consistently hangs on i7 with xen 3.4/3.5 where 2.6.30.3 git kernel works.
No change at all. The xen console is still responding though, so I dumped the full debug output here: http://pastebin.com/d538465af ...as before (XEN) PCI add device 01:03.0 (XEN) PCI add device 01:04.0 (XEN) PCI add device 01:04.2 (XEN) PCI add device 01:04.4 (XEN) PCI add device 01:04.6 (XEN) io_apic.c:2208: (XEN) ioapic_guest_write: apic=0, pin=20, irq=20 (XEN) ioapic_guest_write: new_entry=0001a014 (XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ! (XEN) io_apic.c:2208: (XEN) ioapic_guest_write: apic=0, pin=23, irq=23 (XEN) ioapic_guest_write: new_entry=0001a017 (XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ! (XEN) io_apic.c:2208: (XEN) ioapic_guest_write: apic=0, pin=22, irq=22 (XEN) ioapic_guest_write: new_entry=0001a016 (XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ! (XEN) *** Serial input -> Xen (type ''CTRL-a'' three times to switch input to DOM0) (XEN) ''h'' pressed -> showing installed handlers -Bruce On Sun, Oct 11, 2009 at 11:10 PM, Yu, Ke <ke.yu@intel.com> wrote:> Could you try to add " hpet=disable" option to dom0 command line. If it works, then we meet the same issue. this issue is caused by the dom0 misusing of HPET, we are cooking a patch to fix this issue. > > Best Regards > Ke > > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Bruce Edge > Sent: Sunday, October 11, 2009 10:23 PM > To: xen-devel@lists.xensource.com > Subject: [Xen-devel] xen/master pvops kernel boot consistently hangs on i7 with xen 3.4/3.5 where 2.6.30.3 git kernel works. > > I''m running out of things to try. I would really appreciate a clue as > to what to look at next. It''s not a full lockup as the CTRL-A*3 xen > debug console still works, but the xen/master dom0 shows no signs of > life. > > The current 2.6.31.1 pvops xen/master kernel hangs when running under > Xen on a proliant the a supermicro, both X58 i7 boxes. > > I''ve tried booting with these xen versions (both of which work with a > 2.6.30.3 dom0) > http://xenbits.xensource.com/xen-unstable.hg > http://xenbits.xensource.com/xen-3.4-testing.hg > > And with the 2.6.31.1 source from both (is there a difference?) > git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen > git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git xen/master > > I''ve attached my .config, but here are the xen related settings: > CONFIG_XEN=y > CONFIG_XEN_TIME_VSYSCALL=y > CONFIG_XEN_MAX_DOMAIN_MEMORY=32 > CONFIG_XEN_SAVE_RESTORE=y > CONFIG_XEN_DEBUG_FS=y > CONFIG_XEN_DOM0=y > CONFIG_XEN_PRIVILEGED_GUEST=y > CONFIG_XEN_DOM0_PCI=y > CONFIG_MICROCODE_XEN=y > CONFIG_PCI_XEN=y > CONFIG_XEN_BLKDEV_FRONTEND=y > CONFIG_XEN_NETDEV_FRONTEND=y > CONFIG_XEN_KBDDEV_FRONTEND=m > CONFIG_HVC_XEN=y > CONFIG_XEN_FBDEV_FRONTEND=m > CONFIG_XEN_BALLOON=y > CONFIG_XEN_SCRUB_PAGES=y > CONFIG_XEN_DEV_EVTCHN=y > CONFIG_XEN_BACKEND=y > CONFIG_XEN_BLKDEV_BACKEND=y > CONFIG_XEN_NETDEV_BACKEND=y > CONFIG_XENFS=y > CONFIG_XEN_COMPAT_XENFS=y > CONFIG_XEN_SYS_HYPERVISOR=y > CONFIG_XEN_XENBUS_FRONTEND=y > CONFIG_XEN_S3=y > CONFIG_XEN_GNTDEV=y > > I''m building the kernels with make deb-pkg, installing the resultant 2 > packages, firmware and kernel, and building the initrd. I assume that > that''s no different from running make; make modules_install, etc... on > the target box? > > The boot options I''m using are: > title Xen 3.5-unstable / Ubuntu 9.04, kernel 2.6.31.1 clean > uuid 915b8d5f-dc27-4234-adac-26382dfef7c2 > kernel /xen-3.5-unstable.gz com1=115200,8n1 console=com1 > module /vmlinuz-2.6.31.1 > root=UUID=e0a8abaf-ebb4-4a46-a968-36764653b87f ro console=xvc0 > module /initrd.img-2.6.31.1 > > as well as: > > title Xen 3.5-unstable / Ubuntu 9.04, kernel 2.6.31.1 > uuid 915b8d5f-dc27-4234-adac-26382dfef7c2 > kernel /xen-3.5-unstable.gz com1=115200,8n1 console=com1 > acpi=force apic=on dom0_max_vcpus=1 dom0_vcpus_pin=true > iommu=1,no-intremap,passthrough loglvl=all loglvl_guest=all > module /vmlinuz-2.6.31.1 > root=UUID=e0a8abaf-ebb4-4a46-a968-36764653b87f ro console=xvc0 > pciback.hide=(07:00.0)(07:00.1)(07:00.2)(07:00.3)(14:00.0)(14:00.1)(17:00.0)(17:00.1) > acpi=force > module /initrd.img-2.6.31.1 > > and everything in between. > > I''ve tried booting the xen/master 2.6.31.1 kernel on bare metal > instead of under xen and there''s zero console output. > > Should this work, or does this kernel _only_ run under Xen? > > I can boot both xen versions, 3.5-unstable and 3.4-testing with a > 2.6.30.3 dom0. This _does_ work, but it uses the older kernel: > > title Xen 3.5-unstable / Ubuntu 9.04, kernel 2.6.30.3 > uuid 915b8d5f-dc27-4234-adac-26382dfef7c2 > kernel /xen-3.5-unstable.gz com1=115200,8n1 console=com1 > acpi=force apic=on dom0_max_vcpus=1 dom0_vcpus_pin=true > iommu=1,no-intremap,passthrough loglvl=all loglvl_guest=all > module /vmlinuz-2.6.30.3 > root=UUID=e0a8abaf-ebb4-4a46-a968-36764653b87f ro console=xvc0 > pciback.hide=(07:00.0)(07:00.1)(07:00.2)(07:00.3)(14:00.0)(14:00.1)(17:00.0)(17:00.1) > acpi=force > module /initrd.img-2.6.30.3 > > I''ve tried toggling these settings in the BIOS with no effect: > - hyperthreading > - exec memory protection > - disabling all unneeded PCI cards > - APIC > > Any other BIOS items I should tweak to test? > > This is the tail of the console output before the hang > > (XEN) PCI add device 17:00.0 > (XEN) PCI add device 17:00.1 > (XEN) PCI add device 07:00.0 > (XEN) PCI add device 07:00.1 > (XEN) PCI add device 07:00.2 > (XEN) PCI add device 07:00.3 > (XEN) PCI add device 02:00.0 > (XEN) PCI add device 02:00.1 > (XEN) PCI add device 03:00.0 > (XEN) PCI add device 03:00.1 > (XEN) PCI add device 01:03.0 > (XEN) PCI add device 01:04.0 > (XEN) PCI add device 01:04.2 > (XEN) PCI add device 01:04.4 > (XEN) PCI add device 01:04.6 > (XEN) io_apic.c:2208: > (XEN) ioapic_guest_write: apic=0, pin=20, irq=20 > (XEN) ioapic_guest_write: new_entry=0001a014 > (XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ! > (XEN) io_apic.c:2208: > (XEN) ioapic_guest_write: apic=0, pin=23, irq=23 > (XEN) ioapic_guest_write: new_entry=0001a017 > (XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ! > (XEN) io_apic.c:2208: > (XEN) ioapic_guest_write: apic=0, pin=22, irq=22 > (XEN) ioapic_guest_write: new_entry=0001a016 > (XEN) ioapic_guest_write: Attempt to modify IO-APIC pin for in-use IRQ! > (XEN) *** Serial input -> Xen (type ''CTRL-a'' three times to switch > input to DOM0) > > The CTRL-a does still work. What output from that can I post to help debug this? > > I''ve already posted the full boot output so I''ll resist the temptation > to include it again. > > -Bruce >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel