Pasi Kärkkäinen
2010-Mar-04 19:53 UTC
[Xen-devel] xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
Hello, 32bit PAE 2.6.32.9 xen/stable dom0 kernel crashes early on my (old) testbox.. http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-xen-stable-2.6.32.9-32b-crash-log01.txt ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1]) ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1]) ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0]) (XEN) mm.c:1746:d0 Bad L1 flags 80000100 (XEN) mm.c:778:d0 Bad L1 flags 80000100 (XEN) mm.c:4637:d0 ptwr_emulate: could not get_page_from_l1e() (XEN) d0:v0: unhandled page fault (ec=0003) (XEN) Pagetable walk from c0259fd0: (XEN) L3[0x003] = 000000003c912001 00000912 (XEN) L2[0x001] = 000000003d3b6067 000013b6 (XEN) L1[0x059] = 000000003c259061 00000259 (XEN) domain_crash_sync called from entry.S (ff1cb962) (XEN) Domain 0 (vcpu#0) crashed on cpu#0: (XEN) ----[ Xen-4.0.0-rc5 x86_32p debug=y Not tainted ]---- (XEN) CPU: 0 (XEN) EIP: e019:[<c0405dbe>] (XEN) EFLAGS: 00000296 EM: 1 CONTEXT: pv guest (XEN) eax: 00000000 ebx: c0259fd0 ecx: 80000000 edx: 3c913163 (XEN) esi: 80000000 edi: 00000000 ebp: c0848e54 esp: c0848e34 (XEN) cr0: 8005003b cr4: 000006f0 cr3: 00bc6c80 cr2: c0259fd0 (XEN) ds: e021 es: e021 fs: 00d8 gs: 0000 ss: e021 cs: e019 (XEN) Guest stack trace from esp=c0848e34: (XEN) 00000003 c0405dbe 0001e019 00010096 3c913163 deadbeef deadbeef f57fa000 (XEN) c0848e74 c0405f1c c0259fd0 80000000 3c913163 80000000 c0259fd0 f57fa000 (XEN) c0848e98 c04281f9 3c913163 80000000 80000000 00913163 00005000 f57ff000 (XEN) 0000017b c0848ea8 c042747f 00000005 80000000 c0848ec4 c0404f98 c07c3370 (XEN) c0848ec4 00000000 00000000 00000004 c0848ee0 c089aeec 0000017b 80000000 (XEN) f55002cc f55002cc c07c3370 c0848eec c0895f7a 00000001 c0848f10 c08ad384 (XEN) f5500310 00000001 00000090 f5500280 00000000 00000000 c12aea00 c0848f20 (XEN) c08ad3fa c0895f4d 00000040 c0848f30 c089650f 168ff000 2d1fe000 c0848fac (XEN) c088f56a 00000000 00000000 012aea00 00000000 00000000 00ad5000 00000000 (XEN) c0848f60 00000092 00000000 2d1fe000 00000000 40000000 00000000 c0945be4 (XEN) 00000000 00000000 00000000 00000000 c085ab8c c0848f9c c04063b6 00000000 (XEN) 00000000 00000000 00000000 00000000 00000000 c085ab8c c0848fc4 c088a5bd (XEN) c07b9c5c c06e8010 c08bfb44 00ad5000 c0848fd4 c088a0a8 00ad5000 c08be930 (XEN) c0848ffc c088d113 1fc89375 80000401 00010800 00000f34 00000001 00000000 (XEN) c13af000 00000000 00000000 3d3b1001 00000000 3d3b2001 00000000 3d3b3001 (XEN) 00000000 3c912001 00000000 00000000 00000000 00000000 00000000 00000000 (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 (XEN) Domain 0 crashed: rebooting machine in 5 seconds. (gdb) x/i 0xc0405dbe 0xc0405dbe <xen_set_pte+137>: mov %edx,(%ebx) (gdb) list *0xc0405dbe 0xc0405dbe is in xen_set_pte (arch/x86/xen/mmu.c:703). 698 ADD_STATS(pte_update_batched, paravirt_get_lazy_mode() == PARAVIRT_LAZY_MMU); 699 700 #ifdef CONFIG_X86_PAE 701 ptep->pte_high = pte.pte_high; 702 smp_wmb(); 703 ptep->pte_low = pte.pte_low; 704 #else 705 *ptep = pte; 706 #endif 707 } CONFIG_HIGHPTE is disabled. .config: http://pasik.reaktio.net/xen/pv_ops-dom0-debug/config-2.6.32.9-pvops-dom0-xen-stable-x86_32 -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Mar-04 20:18 UTC
[Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
On 03/04/2010 11:53 AM, Pasi Kärkkäinen wrote:> 32bit PAE 2.6.32.9 xen/stable dom0 kernel crashes early on my (old) testbox.. > > http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-xen-stable-2.6.32.9-32b-crash-log01.txt > > ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) > ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) > ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1]) > ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1]) > ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0]) > (XEN) mm.c:1746:d0 Bad L1 flags 80000100 > (XEN) mm.c:778:d0 Bad L1 flags 80000100 >Does the machine support NX? I''m guessing no. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Mar-04 20:23 UTC
[Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
On Thu, Mar 04, 2010 at 12:18:05PM -0800, Jeremy Fitzhardinge wrote:> On 03/04/2010 11:53 AM, Pasi Kärkkäinen wrote: >> 32bit PAE 2.6.32.9 xen/stable dom0 kernel crashes early on my (old) testbox.. >> >> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-xen-stable-2.6.32.9-32b-crash-log01.txt >> >> ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) >> ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) >> ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1]) >> ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1]) >> ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0]) >> (XEN) mm.c:1746:d0 Bad L1 flags 80000100 >> (XEN) mm.c:778:d0 Bad L1 flags 80000100 >> > > Does the machine support NX? I''m guessing no. >on baremetal: # dmesg | grep NX Using x86 segment limits to approximate NX protection # grep flags /proc/cpuinfo flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe constant_tsc up pebs bts pni dtes64 monitor ds_cpl cid xtpr -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Mar-04 20:28 UTC
Re: [Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
On Thu, Mar 04, 2010 at 10:23:16PM +0200, Pasi Kärkkäinen wrote:> On Thu, Mar 04, 2010 at 12:18:05PM -0800, Jeremy Fitzhardinge wrote: > > On 03/04/2010 11:53 AM, Pasi Kärkkäinen wrote: > >> 32bit PAE 2.6.32.9 xen/stable dom0 kernel crashes early on my (old) testbox.. > >> > >> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-xen-stable-2.6.32.9-32b-crash-log01.txt > >> > >> ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) > >> ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) > >> ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1]) > >> ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1]) > >> ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0]) > >> (XEN) mm.c:1746:d0 Bad L1 flags 80000100 > >> (XEN) mm.c:778:d0 Bad L1 flags 80000100 > >> > > > > Does the machine support NX? I''m guessing no. > > > > on baremetal: > > # dmesg | grep NX > Using x86 segment limits to approximate NX protection > > # grep flags /proc/cpuinfo > flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe constant_tsc up pebs bts pni dtes64 monitor ds_cpl cid xtpr >It was 2.6.31.12-174.2.22.fc12.i686.PAE saying the above.. 2.6.32.9 xen/stable booted on baremetal doesn''t say anything about NX. -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Mar-04 21:33 UTC
Re: [Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
On 03/04/2010 12:28 PM, Pasi Kärkkäinen wrote:> It was 2.6.31.12-174.2.22.fc12.i686.PAE saying the above.. > 2.6.32.9 xen/stable booted on baremetal doesn''t say anything about NX. >What happens if you boot with noexec=no on the kernel command line? J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Mar-05 07:13 UTC
Re: [Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
On Thu, Mar 04, 2010 at 01:33:39PM -0800, Jeremy Fitzhardinge wrote:> On 03/04/2010 12:28 PM, Pasi Kärkkäinen wrote: >> It was 2.6.31.12-174.2.22.fc12.i686.PAE saying the above.. >> 2.6.32.9 xen/stable booted on baremetal doesn''t say anything about NX. >> > > What happens if you boot with noexec=no on the kernel command line? >noexec=no didn''t change anything.. still crashes. console output is the same as earlier. I also checked the BIOS, it didn''t seem to have any NX settings. -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Mar-05 10:51 UTC
Re: [Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
On Fri, Mar 05, 2010 at 09:13:59AM +0200, Pasi Kärkkäinen wrote:> On Thu, Mar 04, 2010 at 01:33:39PM -0800, Jeremy Fitzhardinge wrote: > > On 03/04/2010 12:28 PM, Pasi Kärkkäinen wrote: > >> It was 2.6.31.12-174.2.22.fc12.i686.PAE saying the above.. > >> 2.6.32.9 xen/stable booted on baremetal doesn''t say anything about NX. > >> > > > > What happens if you boot with noexec=no on the kernel command line? > > > > noexec=no didn''t change anything.. still crashes. console output is the same as earlier. > I also checked the BIOS, it didn''t seem to have any NX settings. >latest xen/master 2.6.31.6 seems to boot OK on this box, with the same Xen 4.0.0-rc5 hypervisor. successful bootlog of xen/master 2.6.31.6 here: http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-xen-master-2.6.31.6-32b-works-log03.txt -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Mar-06 15:05 UTC
Re: [Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
On Fri, Mar 05, 2010 at 12:51:02PM +0200, Pasi Kärkkäinen wrote:> On Fri, Mar 05, 2010 at 09:13:59AM +0200, Pasi Kärkkäinen wrote: > > On Thu, Mar 04, 2010 at 01:33:39PM -0800, Jeremy Fitzhardinge wrote: > > > On 03/04/2010 12:28 PM, Pasi Kärkkäinen wrote: > > >> It was 2.6.31.12-174.2.22.fc12.i686.PAE saying the above.. > > >> 2.6.32.9 xen/stable booted on baremetal doesn''t say anything about NX. > > >> > > > > > > What happens if you boot with noexec=no on the kernel command line? > > > > > > > noexec=no didn''t change anything.. still crashes. console output is the same as earlier. > > I also checked the BIOS, it didn''t seem to have any NX settings. > > > > latest xen/master 2.6.31.6 seems to boot OK on this box, with the same Xen 4.0.0-rc5 hypervisor. > > successful bootlog of xen/master 2.6.31.6 here: > http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-xen-master-2.6.31.6-32b-works-log03.txt >xen/stable 2.6.32.9 boots OK on baremetal (without Xen). Does it help to have a log of the boot on baremetal? Here are a couple of xen/stable 2.6.32.9 crashing boots as Xen dom0, with various debugging options, and also with "acpi=off": two logs in the same file: http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-xen-stable-2.6.32.9-32b-crash-log04.txt -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Mar-08 18:17 UTC
Re: [Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
On Sat, Mar 06, 2010 at 05:05:09PM +0200, Pasi Kärkkäinen wrote:> On Fri, Mar 05, 2010 at 12:51:02PM +0200, Pasi Kärkkäinen wrote: > > On Fri, Mar 05, 2010 at 09:13:59AM +0200, Pasi Kärkkäinen wrote: > > > On Thu, Mar 04, 2010 at 01:33:39PM -0800, Jeremy Fitzhardinge wrote: > > > > On 03/04/2010 12:28 PM, Pasi Kärkkäinen wrote: > > > >> It was 2.6.31.12-174.2.22.fc12.i686.PAE saying the above.. > > > >> 2.6.32.9 xen/stable booted on baremetal doesn''t say anything about NX. > > > >> > > > > > > > > What happens if you boot with noexec=no on the kernel command line? > > > > > > > > > > noexec=no didn''t change anything.. still crashes. console output is the same as earlier. > > > I also checked the BIOS, it didn''t seem to have any NX settings. > > > > > > > latest xen/master 2.6.31.6 seems to boot OK on this box, with the same Xen 4.0.0-rc5 hypervisor. > > > > successful bootlog of xen/master 2.6.31.6 here: > > http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-xen-master-2.6.31.6-32b-works-log03.txt > > > > xen/stable 2.6.32.9 boots OK on baremetal (without Xen). > Does it help to have a log of the boot on baremetal? > > Here are a couple of xen/stable 2.6.32.9 crashing boots as Xen dom0, > with various debugging options, and also with "acpi=off":Could you find what is at c0405dbe and around that area with gdb? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Mar-09 18:37 UTC
Re: [Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
On Mon, Mar 08, 2010 at 01:17:13PM -0500, Konrad Rzeszutek Wilk wrote:> On Sat, Mar 06, 2010 at 05:05:09PM +0200, Pasi Kärkkäinen wrote: > > On Fri, Mar 05, 2010 at 12:51:02PM +0200, Pasi Kärkkäinen wrote: > > > On Fri, Mar 05, 2010 at 09:13:59AM +0200, Pasi Kärkkäinen wrote: > > > > On Thu, Mar 04, 2010 at 01:33:39PM -0800, Jeremy Fitzhardinge wrote: > > > > > On 03/04/2010 12:28 PM, Pasi Kärkkäinen wrote: > > > > >> It was 2.6.31.12-174.2.22.fc12.i686.PAE saying the above.. > > > > >> 2.6.32.9 xen/stable booted on baremetal doesn''t say anything about NX. > > > > >> > > > > > > > > > > What happens if you boot with noexec=no on the kernel command line? > > > > > > > > > > > > > noexec=no didn''t change anything.. still crashes. console output is the same as earlier. > > > > I also checked the BIOS, it didn''t seem to have any NX settings. > > > > > > > > > > latest xen/master 2.6.31.6 seems to boot OK on this box, with the same Xen 4.0.0-rc5 hypervisor. > > > > > > successful bootlog of xen/master 2.6.31.6 here: > > > http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-xen-master-2.6.31.6-32b-works-log03.txt > > > > > > > xen/stable 2.6.32.9 boots OK on baremetal (without Xen). > > Does it help to have a log of the boot on baremetal? > > > > Here are a couple of xen/stable 2.6.32.9 crashing boots as Xen dom0, > > with various debugging options, and also with "acpi=off": > > Could you find what is at c0405dbe and around that area with gdb? >I''m away from that computer for a couple of days now.. I can check that when I have access to it again. In the first email [1] of this thread I had some gdb stuff, does that help? -- Pasi [1] http://lists.xensource.com/archives/html/xen-devel/2010-03/msg00243.html _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Mar-09 19:33 UTC
Re: [Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
On 03/08/2010 10:17 AM, Konrad Rzeszutek Wilk wrote:> On Sat, Mar 06, 2010 at 05:05:09PM +0200, Pasi Kärkkäinen wrote: > >> On Fri, Mar 05, 2010 at 12:51:02PM +0200, Pasi Kärkkäinen wrote: >> >>> On Fri, Mar 05, 2010 at 09:13:59AM +0200, Pasi Kärkkäinen wrote: >>> >>>> On Thu, Mar 04, 2010 at 01:33:39PM -0800, Jeremy Fitzhardinge wrote: >>>> >>>>> On 03/04/2010 12:28 PM, Pasi Kärkkäinen wrote: >>>>> >>>>>> It was 2.6.31.12-174.2.22.fc12.i686.PAE saying the above.. >>>>>> 2.6.32.9 xen/stable booted on baremetal doesn''t say anything about NX. >>>>>> >>>>>> >>>>> What happens if you boot with noexec=no on the kernel command line? >>>>> >>>>> >>>> noexec=no didn''t change anything.. still crashes. console output is the same as earlier. >>>> I also checked the BIOS, it didn''t seem to have any NX settings. >>>> >>>> >>> latest xen/master 2.6.31.6 seems to boot OK on this box, with the same Xen 4.0.0-rc5 hypervisor. >>> >>> successful bootlog of xen/master 2.6.31.6 here: >>> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-xen-master-2.6.31.6-32b-works-log03.txt >>> >>> >> xen/stable 2.6.32.9 boots OK on baremetal (without Xen). >> Does it help to have a log of the boot on baremetal? >> >> Here are a couple of xen/stable 2.6.32.9 crashing boots as Xen dom0, >> with various debugging options, and also with "acpi=off": >> > Could you find what is at c0405dbe and around that area with gdb? >I think the problem is that there''s something happening in kernel/head_32.S to test for NX and disable it if not present, which isn''t happening on the Xen startup path. But I haven''t investigated that yet. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Mar-18 22:59 UTC
Re: [Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
On Tue, Mar 09, 2010 at 11:33:18AM -0800, Jeremy Fitzhardinge wrote:> On 03/08/2010 10:17 AM, Konrad Rzeszutek Wilk wrote: >> On Sat, Mar 06, 2010 at 05:05:09PM +0200, Pasi Kärkkäinen wrote: >> >>> On Fri, Mar 05, 2010 at 12:51:02PM +0200, Pasi Kärkkäinen wrote: >>> >>>> On Fri, Mar 05, 2010 at 09:13:59AM +0200, Pasi Kärkkäinen wrote: >>>> >>>>> On Thu, Mar 04, 2010 at 01:33:39PM -0800, Jeremy Fitzhardinge wrote: >>>>> >>>>>> On 03/04/2010 12:28 PM, Pasi Kärkkäinen wrote: >>>>>> >>>>>>> It was 2.6.31.12-174.2.22.fc12.i686.PAE saying the above.. >>>>>>> 2.6.32.9 xen/stable booted on baremetal doesn''t say anything about NX. >>>>>>> >>>>>>> >>>>>> What happens if you boot with noexec=no on the kernel command line? >>>>>> >>>>>> >>>>> noexec=no didn''t change anything.. still crashes. console output is the same as earlier. >>>>> I also checked the BIOS, it didn''t seem to have any NX settings. >>>>> >>>>> >>>> latest xen/master 2.6.31.6 seems to boot OK on this box, with the same Xen 4.0.0-rc5 hypervisor. >>>> >>>> successful bootlog of xen/master 2.6.31.6 here: >>>> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-xen-master-2.6.31.6-32b-works-log03.txt >>>> >>>> >>> xen/stable 2.6.32.9 boots OK on baremetal (without Xen). >>> Does it help to have a log of the boot on baremetal? >>> >>> Here are a couple of xen/stable 2.6.32.9 crashing boots as Xen dom0, >>> with various debugging options, and also with "acpi=off": >>> >> Could you find what is at c0405dbe and around that area with gdb? >> > > I think the problem is that there''s something happening in > kernel/head_32.S to test for NX and disable it if not present, which > isn''t happening on the Xen startup path. But I haven''t investigated > that yet. >Should I try using kernel/head_32.S from 2.6.31.6 with 2.6.32.10 dom0 kernel? I have access to the machine in question again on sunday, so I could test things then. -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Mar-18 23:25 UTC
Re: [Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
On 03/18/2010 03:59 PM, Pasi Kärkkäinen wrote:> Should I try using kernel/head_32.S from 2.6.31.6 with 2.6.32.10 dom0 kernel? > I have access to the machine in question again on sunday, so I could test things then. >It looks like there''s no significant difference between them. I''m at a bit of a loss really; "noexec=off" really should have avoided the problem if it is NX. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Mar-18 23:32 UTC
[Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
On 03/04/2010 11:53 AM, Pasi Kärkkäinen wrote:> Hello, > > 32bit PAE 2.6.32.9 xen/stable dom0 kernel crashes early on my (old) testbox.. > > http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-xen-stable-2.6.32.9-32b-crash-log01.txt > > ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) > ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) > ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1]) > ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1]) > ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0]) > (XEN) mm.c:1746:d0 Bad L1 flags 80000100 > (XEN) mm.c:778:d0 Bad L1 flags 80000100 > (XEN) mm.c:4637:d0 ptwr_emulate: could not get_page_from_l1e() > (XEN) d0:v0: unhandled page fault (ec=0003) > (XEN) Pagetable walk from c0259fd0: > (XEN) L3[0x003] = 000000003c912001 00000912 > (XEN) L2[0x001] = 000000003d3b6067 000013b6 > (XEN) L1[0x059] = 000000003c259061 00000259 > (XEN) domain_crash_sync called from entry.S (ff1cb962) > (XEN) Domain 0 (vcpu#0) crashed on cpu#0: > (XEN) ----[ Xen-4.0.0-rc5 x86_32p debug=y Not tainted ]---- > (XEN) CPU: 0 > (XEN) EIP: e019:[<c0405dbe>] > (XEN) EFLAGS: 00000296 EM: 1 CONTEXT: pv guest > (XEN) eax: 00000000 ebx: c0259fd0 ecx: 80000000 edx: 3c913163 > (XEN) esi: 80000000 edi: 00000000 ebp: c0848e54 esp: c0848e34 > (XEN) cr0: 8005003b cr4: 000006f0 cr3: 00bc6c80 cr2: c0259fd0 > (XEN) ds: e021 es: e021 fs: 00d8 gs: 0000 ss: e021 cs: e019 > (XEN) Guest stack trace from esp=c0848e34: > (XEN) 00000003 c0405dbe 0001e019 00010096 3c913163 deadbeef deadbeef f57fa000 > (XEN) c0848e74 c0405f1c c0259fd0 80000000 3c913163 80000000 c0259fd0 f57fa000 > (XEN) c0848e98 c04281f9 3c913163 80000000 80000000 00913163 00005000 f57ff000 > (XEN) 0000017b c0848ea8 c042747f 00000005 80000000 c0848ec4 c0404f98 c07c3370 > (XEN) c0848ec4 00000000 00000000 00000004 c0848ee0 c089aeec 0000017b 80000000 > (XEN) f55002cc f55002cc c07c3370 c0848eec c0895f7a 00000001 c0848f10 c08ad384 > (XEN) f5500310 00000001 00000090 f5500280 00000000 00000000 c12aea00 c0848f20 > (XEN) c08ad3fa c0895f4d 00000040 c0848f30 c089650f 168ff000 2d1fe000 c0848fac > (XEN) c088f56a 00000000 00000000 012aea00 00000000 00000000 00ad5000 00000000 > (XEN) c0848f60 00000092 00000000 2d1fe000 00000000 40000000 00000000 c0945be4 > (XEN) 00000000 00000000 00000000 00000000 c085ab8c c0848f9c c04063b6 00000000 > (XEN) 00000000 00000000 00000000 00000000 00000000 c085ab8c c0848fc4 c088a5bd > (XEN) c07b9c5c c06e8010 c08bfb44 00ad5000 c0848fd4 c088a0a8 00ad5000 c08be930 > (XEN) c0848ffc c088d113 1fc89375 80000401 00010800 00000f34 00000001 00000000 > (XEN) c13af000 00000000 00000000 3d3b1001 00000000 3d3b2001 00000000 3d3b3001 > (XEN) 00000000 3c912001 00000000 00000000 00000000 00000000 00000000 00000000 > (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 > (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 > (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 > (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 > (XEN) Domain 0 crashed: rebooting machine in 5 seconds. > > > > (gdb) x/i 0xc0405dbe > 0xc0405dbe<xen_set_pte+137>: mov %edx,(%ebx) > > (gdb) list *0xc0405dbe > 0xc0405dbe is in xen_set_pte (arch/x86/xen/mmu.c:703). > 698 ADD_STATS(pte_update_batched, paravirt_get_lazy_mode() == PARAVIRT_LAZY_MMU); > 699 > 700 #ifdef CONFIG_X86_PAE > 701 ptep->pte_high = pte.pte_high; > 702 smp_wmb(); > 703 ptep->pte_low = pte.pte_low; > 704 #else > 705 *ptep = pte; > 706 #endif > 707 } >Xen is 32 bit, right? Could you disassemble this whole function, so I can see what register the other half is in? (I''m guessing its ecx or esi; nothing else makes much sense.) Could you also try to work out at least some of the call stack to see what is trying to set a pte? The presence of _PAGE_UNUSED1 suggests its the CPA test. If so, you could try disabling that (CONFIG_CPA_DEBUG). Keir, 32-bit Xen isn''t reserving bit 9 in the PTE is it? It just uses the high user bits? Thanks, J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Mar-18 23:37 UTC
[Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
On Thu, Mar 18, 2010 at 04:32:10PM -0700, Jeremy Fitzhardinge wrote:> On 03/04/2010 11:53 AM, Pasi Kärkkäinen wrote: >> Hello, >> >> 32bit PAE 2.6.32.9 xen/stable dom0 kernel crashes early on my (old) testbox.. >> >> http://pasik.reaktio.net/xen/pv_ops-dom0-debug/pv_ops-dom0-xen-stable-2.6.32.9-32b-crash-log01.txt >> >> ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) >> ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) >> ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1]) >> ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1]) >> ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0]) >> (XEN) mm.c:1746:d0 Bad L1 flags 80000100 >> (XEN) mm.c:778:d0 Bad L1 flags 80000100 >> (XEN) mm.c:4637:d0 ptwr_emulate: could not get_page_from_l1e() >> (XEN) d0:v0: unhandled page fault (ec=0003) >> (XEN) Pagetable walk from c0259fd0: >> (XEN) L3[0x003] = 000000003c912001 00000912 >> (XEN) L2[0x001] = 000000003d3b6067 000013b6 >> (XEN) L1[0x059] = 000000003c259061 00000259 >> (XEN) domain_crash_sync called from entry.S (ff1cb962) >> (XEN) Domain 0 (vcpu#0) crashed on cpu#0: >> (XEN) ----[ Xen-4.0.0-rc5 x86_32p debug=y Not tainted ]---- >> (XEN) CPU: 0 >> (XEN) EIP: e019:[<c0405dbe>] >> (XEN) EFLAGS: 00000296 EM: 1 CONTEXT: pv guest >> (XEN) eax: 00000000 ebx: c0259fd0 ecx: 80000000 edx: 3c913163 >> (XEN) esi: 80000000 edi: 00000000 ebp: c0848e54 esp: c0848e34 >> (XEN) cr0: 8005003b cr4: 000006f0 cr3: 00bc6c80 cr2: c0259fd0 >> (XEN) ds: e021 es: e021 fs: 00d8 gs: 0000 ss: e021 cs: e019 >> (XEN) Guest stack trace from esp=c0848e34: >> (XEN) 00000003 c0405dbe 0001e019 00010096 3c913163 deadbeef deadbeef f57fa000 >> (XEN) c0848e74 c0405f1c c0259fd0 80000000 3c913163 80000000 c0259fd0 f57fa000 >> (XEN) c0848e98 c04281f9 3c913163 80000000 80000000 00913163 00005000 f57ff000 >> (XEN) 0000017b c0848ea8 c042747f 00000005 80000000 c0848ec4 c0404f98 c07c3370 >> (XEN) c0848ec4 00000000 00000000 00000004 c0848ee0 c089aeec 0000017b 80000000 >> (XEN) f55002cc f55002cc c07c3370 c0848eec c0895f7a 00000001 c0848f10 c08ad384 >> (XEN) f5500310 00000001 00000090 f5500280 00000000 00000000 c12aea00 c0848f20 >> (XEN) c08ad3fa c0895f4d 00000040 c0848f30 c089650f 168ff000 2d1fe000 c0848fac >> (XEN) c088f56a 00000000 00000000 012aea00 00000000 00000000 00ad5000 00000000 >> (XEN) c0848f60 00000092 00000000 2d1fe000 00000000 40000000 00000000 c0945be4 >> (XEN) 00000000 00000000 00000000 00000000 c085ab8c c0848f9c c04063b6 00000000 >> (XEN) 00000000 00000000 00000000 00000000 00000000 c085ab8c c0848fc4 c088a5bd >> (XEN) c07b9c5c c06e8010 c08bfb44 00ad5000 c0848fd4 c088a0a8 00ad5000 c08be930 >> (XEN) c0848ffc c088d113 1fc89375 80000401 00010800 00000f34 00000001 00000000 >> (XEN) c13af000 00000000 00000000 3d3b1001 00000000 3d3b2001 00000000 3d3b3001 >> (XEN) 00000000 3c912001 00000000 00000000 00000000 00000000 00000000 00000000 >> (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 >> (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 >> (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 >> (XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 >> (XEN) Domain 0 crashed: rebooting machine in 5 seconds. >> >> >> >> (gdb) x/i 0xc0405dbe >> 0xc0405dbe<xen_set_pte+137>: mov %edx,(%ebx) >> >> (gdb) list *0xc0405dbe >> 0xc0405dbe is in xen_set_pte (arch/x86/xen/mmu.c:703). >> 698 ADD_STATS(pte_update_batched, paravirt_get_lazy_mode() == PARAVIRT_LAZY_MMU); >> 699 >> 700 #ifdef CONFIG_X86_PAE >> 701 ptep->pte_high = pte.pte_high; >> 702 smp_wmb(); >> 703 ptep->pte_low = pte.pte_low; >> 704 #else >> 705 *ptep = pte; >> 706 #endif >> 707 } >> > > Xen is 32 bit, right? >Yes, this is an old box that is not 64bit capable. It''s running 32bit PAE Xen 4.0.0-rc6 hypervisor.> Could you disassemble this whole function, so I can see what register > the other half is in? (I''m guessing its ecx or esi; nothing else makes > much sense.) > > Could you also try to work out at least some of the call stack to see > what is trying to set a pte? The presence of _PAGE_UNUSED1 suggests its > the CPA test. If so, you could try disabling that (CONFIG_CPA_DEBUG). >Ok, I''ll check these when I have access to the box again. -- Pasi> Keir, 32-bit Xen isn''t reserving bit 9 in the PTE is it? It just uses > the high user bits? > > Thanks, > J_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
M A Young
2010-Mar-19 22:20 UTC
Re: [Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
On Thu, 18 Mar 2010, Jeremy Fitzhardinge wrote:>> (gdb) x/i 0xc0405dbe >> 0xc0405dbe<xen_set_pte+137>: mov %edx,(%ebx) >> >> (gdb) list *0xc0405dbe >> 0xc0405dbe is in xen_set_pte (arch/x86/xen/mmu.c:703). >> 698 ADD_STATS(pte_update_batched, paravirt_get_lazy_mode() == >> PARAVIRT_LAZY_MMU); >> 699 >> 700 #ifdef CONFIG_X86_PAE >> 701 ptep->pte_high = pte.pte_high; >> 702 smp_wmb(); >> 703 ptep->pte_low = pte.pte_low; >> 704 #else >> 705 *ptep = pte; >> 706 #endif >> 707 } >> > > Xen is 32 bit, right? > > Could you disassemble this whole function, so I can see what register the > other half is in? (I''m guessing its ecx or esi; nothing else makes much > sense.)I submitted an earlier report with a more complete dump of this function, which is at http://lists.xensource.com/archives/html/xen-devel/2010-02/msg00844.html The code may have changed slightly since then but it looks to be the same issue. Michael Young _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Mar-19 22:57 UTC
Re: [Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
On 03/19/2010 03:20 PM, M A Young wrote:> >> Xen is 32 bit, right? >> >> Could you disassemble this whole function, so I can see what register >> the other half is in? (I''m guessing its ecx or esi; nothing else >> makes much sense.) > > > I submitted an earlier report with a more complete dump of this > function, which is at > http://lists.xensource.com/archives/html/xen-devel/2010-02/msg00844.html > The code may have changed slightly since then but it looks to be the > same issue.I think so too. The question is where that set_pte is coming from, and why does it have NX set even though the CPU doesn''t support it? I''m setting up a Xen test domain at the moment, so I''ll have a look into it. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Mar-20 01:00 UTC
Re: [Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
On 03/19/2010 03:20 PM, M A Young wrote:> I submitted an earlier report with a more complete dump of this > function, which is at > http://lists.xensource.com/archives/html/xen-devel/2010-02/msg00844.html > The code may have changed slightly since then but it looks to be the > same issue.I can reproduce this with 32-bit Xen even on a system supporting NX (below). Due to a brainfart I had been reading it as bit 9 (for programmer use), but its actually Global, which we don''t allow. Not sure why Global is leaking through; I''ll have a look later. (With luck it will also explain why NX is leaking through; probably a path which isn''t masking the supported pte flags). J [ 0.000000] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x18] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x1a] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x1c] disabled) [ 0.000000] ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0]) (XEN) mm.c:1746:d0 Bad L1 flags 100 (XEN) mm.c:778:d0 Bad L1 flags 100 (XEN) mm.c:4637:d0 ptwr_emulate: could not get_page_from_l1e() (XEN) d0:v0: unhandled page fault (ec=0003) (XEN) Pagetable walk from c17d9fd0: (XEN) L3[0x003] = 00000000197d7001 000017d7 (XEN) L2[0x00b] = 000000001a3c8067 000023c8 (XEN) L1[0x1d9] = 80000000197d9061 000017d9 (XEN) domain_crash_sync called from entry.S (ff1cb682) (XEN) Domain 0 (vcpu#0) crashed on cpu#0: (XEN) ----[ Xen-4.0.0-rc6 x86_32p debug=y Not tainted ]---- (XEN) CPU: 0 (XEN) EIP: e019:[<c1005ed9>] (XEN) EFLAGS: 00000296 EM: 1 CONTEXT: pv guest (XEN) eax: 00000000 ebx: c17d9fd0 ecx: 80000000 edx: 197d8163 (XEN) esi: 80000000 edi: 00000000 ebp: c16d3e54 esp: c16d3e34 (XEN) cr0: 8005003b cr4: 000006f0 cr3: 003f4c80 cr2: c17d9fd0 (XEN) ds: e021 es: e021 fs: 00d8 gs: 0000 ss: e021 cs: e019 (XEN) Guest stack trace from esp=c16d3e34: (XEN) 00000003 c1005ed9 0001e019 00010096 197d8163 deadbeef deadbeef f57fa000 (XEN) c16d3e74 c1006037 c17d9fd0 80000000 197d8163 80000000 c17d9fd0 f57fa000 (XEN) c16d3e98 c10281c5 197d8163 80000000 80000000 017d8163 00005000 f57ff000 (XEN) 0000017b c16d3ea8 c10273ab 00000005 80000000 c16d3ec4 c1005058 c160fa5f (XEN) c16d3ec4 00000000 00000000 00000001 c16d3ee0 c1766642 0000017b 80000000 (XEN) f5600c24 f5600c24 c160fa5f c16d3eec c1761b92 00000001 c16d3f10 c1778ad9 (XEN) f5600ca8 00000001 000000d8 f5600bd0 00000000 00000000 00000000 c16d3f20 (XEN) c1778b4f c1761b65 00000040 c16d3f30 c1762127 1d010000 00000000 c16d3fac (XEN) c175c965 c17e8dac c103a2f1 02341600 00000000 00000035 019b8000 00000000 (XEN) c16d3f6d 00000000 205b8000 00989600 3030302e 1d010000 00000000 c16e10ac (XEN) 00000002 00000002 c16d3f8c 00000000 00000000 c16e034c c16d3f9c 00000000 (XEN) 00000000 00000000 00000000 00000000 00000000 c16e034c c16d3fc4 c17575bd (XEN) c16078d7 c1494010 c178e90c 019b8000 c16d3fd4 c17570a8 019b8000 c178d6f8 (XEN) c16d3ffc c175a26b 17889375 80980201 00200800 000106e5 00000001 00000000 (XEN) c23b7000 00000000 00000000 (XEN) Domain 0 crashed: rebooting machine in 5 seconds. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Mar-23 04:03 UTC
Re: [Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
On 03/19/2010 03:20 PM, M A Young wrote:>> Xen is 32 bit, right? >> >> Could you disassemble this whole function, so I can see what register >> the other half is in? (I''m guessing its ecx or esi; nothing else >> makes much sense.) > > > I submitted an earlier report with a more complete dump of this > function, which is at > http://lists.xensource.com/archives/html/xen-devel/2010-02/msg00844.html > The code may have changed slightly since then but it looks to be the > same issue.OK, I just committed a fix for this. The IOAPIC dummy mapping wasn''t being created properly, and it was failing to mask out the unsupported PTE flags. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Mar-23 07:02 UTC
Re: [Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot
On Mon, Mar 22, 2010 at 09:03:18PM -0700, Jeremy Fitzhardinge wrote:> On 03/19/2010 03:20 PM, M A Young wrote: >>> Xen is 32 bit, right? >>> >>> Could you disassemble this whole function, so I can see what register >>> the other half is in? (I''m guessing its ecx or esi; nothing else >>> makes much sense.) >> >> >> I submitted an earlier report with a more complete dump of this >> function, which is at >> http://lists.xensource.com/archives/html/xen-devel/2010-02/msg00844.html >> The code may have changed slightly since then but it looks to be the >> same issue. > > OK, I just committed a fix for this. The IOAPIC dummy mapping wasn''t > being created properly, and it was failing to mask out the unsupported > PTE flags. >Great! I''ll try it. -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Mar-23 11:18 UTC
Re: [Xen-devel] Re: xen/stable 2.6.32.9 32bit dom0 kernel crashes early on boot / works now!
On Mon, Mar 22, 2010 at 09:03:18PM -0700, Jeremy Fitzhardinge wrote:> On 03/19/2010 03:20 PM, M A Young wrote: >>> Xen is 32 bit, right? >>> >>> Could you disassemble this whole function, so I can see what register >>> the other half is in? (I''m guessing its ecx or esi; nothing else >>> makes much sense.) >> >> >> I submitted an earlier report with a more complete dump of this >> function, which is at >> http://lists.xensource.com/archives/html/xen-devel/2010-02/msg00844.html >> The code may have changed slightly since then but it looks to be the >> same issue. > > OK, I just committed a fix for this. The IOAPIC dummy mapping wasn''t > being created properly, and it was failing to mask out the unsupported > PTE flags. >And yes, this patch fixes the problem. 2.6.32.10 32bit PAE dom0 from xen/stable-2.6.32.x now boots OK for me! Thanks! -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel