Hi Keir, For me the current unstable tree is not working with 8 cpu and SMP dom0 on a IA32 server. I found that this patch http://xenbits.xensource.com/xen-unstable.hg?cmd=changeset;node=a0b4156c c0ce4e6495cfa316c5383ee2be81c22f has introduced some issue. I also see same issue if I pass maxcpus=4 option in the hypervisor command line. After reverting the patch, SMP dom0 can boot with 8 cpus with the 6453 release from the xen-unstable. I see these errors while booting the SMP dom0 kernel. NET: Registered protocol family 16 PCI: Using MMCONFIG ACPI: Subsystem revision 20050309 ACPI: Interpreter enabled ACPI: Using IOAPIC for interrupt routing ACPI: PCI Root Bridge [TNB] (0000:00) PCI: Probing PCI hardware (bus 00) PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.2 (XEN) (file=/home/nitin/repo/hariwch/latest/xen/include/asm/mm.h, line=201) Error pfn 0: rd=ffbf4a00, od=ff1a5080, caf=80000001, taf=f0000001 (XEN) DOM0: (file=mm.c, line=349) Could not get page ref for pfn 0 (XEN) DOM0: (file=mm.c, line=2085) Could not get page for normal update (XEN) DOM0: (file=mm.c, line=2243) Invalid page update command 2 (XEN) DOM0: (file=mm.c, line=349) Could not get page ref for pfn 23d201 (XEN) DOM0: (file=mm.c, line=2085) Could not get page for normal update (XEN) DOM0: (file=mm.c, line=349) Could not get page ref for pfn 693d200 (XEN) DOM0: (file=mm.c, line=2085) Could not get page for normal update (XEN) DOM0: (file=mm.c, line=349) Could not get page ref for pfn 2fbf7d (XEN) DOM0: (file=mm.c, line=2218) Could not get page for mach->phys update (XEN) DOM0: (file=mm.c, line=2243) Invalid page update command 3 (XEN) DOM0: (file=mm.c, line=2243) Invalid page update command 2 (XEN) DOM0: (file=mm.c, line=349) Could not get page ref for pfn 28200010 (XEN) DOM0: (file=mm.c, line=2218) Could not get page for mach->phys update (XEN) DOM0: (file=mm.c, line=2243) Invalid page update command 2 (XEN) DOM0: (file=mm.c, line=349) Could not get page ref for pfn ab000000 Thanks & Regards, Nitin ------------------------------------------------------------------------ ----------- Sr Software Engineer Open Source Technology Center, Intel Corp _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> For me the current unstable tree is not working with 8 cpu and SMP > dom0 on a IA32 server. I found that this patch > http://xenbits.xensource.com/xen-unstable.hg?cmd=changeset; > node=a0b4156cc0ce4e6495cfa316c5383ee2be81c22f has introduced some > issue. I also see same issue if I pass maxcpus=4 option in the > hypervisor command line.The errors you see with the patch reverted, the fact you can reproduce by changing maxcpus, and the also because the patch itself is pretty benign, all these problems must be caused by some underlying memory corruption. One possibility is that it''s due to my fiddling with the domain 0 memory map (I now free pseudophys range 0-1MB to the linux memoy allocator, which we didn''''t previously do). This might mean that we''re now allocating memory that gets scribbled on.... It might be worth explicitly unmapping the low pseudophys 1MB, not free it to the memory allocator, and then see if you can boot dom0. Then we can see if we have any code paths that go straight at low memory, which is a bug on xenlinux. -- Keir> After reverting the patch, SMP dom0 can boot with 8 cpus with the 6453 > release from the xen-unstable. > > I see these errors while booting the SMP dom0 kernel. > > NET: Registered protocol family 16 > PCI: Using MMCONFIG > ACPI: Subsystem revision 20050309 > ACPI: Interpreter enabled > ACPI: Using IOAPIC for interrupt routing > ACPI: PCI Root Bridge [TNB] (0000:00) > PCI: Probing PCI hardware (bus 00) > PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.2 > (XEN) (file=/home/nitin/repo/hariwch/latest/xen/include/asm/mm.h, > line=201) Error pfn 0: rd=ffbf4a00, od=ff1a5080, caf=80000001, > taf=f0000001 > (XEN) DOM0: (file=mm.c, line=349) Could not get page ref for pfn 0 > (XEN) DOM0: (file=mm.c, line=2085) Could not get page for normal update > (XEN) DOM0: (file=mm.c, line=2243) Invalid page update command 2 > (XEN) DOM0: (file=mm.c, line=349) Could not get page ref for pfn 23d201 > (XEN) DOM0: (file=mm.c, line=2085) Could not get page for normal update > (XEN) DOM0: (file=mm.c, line=349) Could not get page ref for pfn > 693d200 > (XEN) DOM0: (file=mm.c, line=2085) Could not get page for normal update > (XEN) DOM0: (file=mm.c, line=349) Could not get page ref for pfn 2fbf7d > (XEN) DOM0: (file=mm.c, line=2218) Could not get page for mach->phys > update > (XEN) DOM0: (file=mm.c, line=2243) Invalid page update command 3 > (XEN) DOM0: (file=mm.c, line=2243) Invalid page update command 2 > (XEN) DOM0: (file=mm.c, line=349) Could not get page ref for pfn > 28200010 > (XEN) DOM0: (file=mm.c, line=2218) Could not get page for mach->phys > update > (XEN) DOM0: (file=mm.c, line=2243) Invalid page update command 2 > (XEN) DOM0: (file=mm.c, line=349) Could not get page ref for pfn > ab000000 > > > Thanks & Regards, > Nitin > ----------------------------------------------------------------------- > ------------ > Sr Software Engineer > Open Source Technology Center, Intel Corp >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> One possibility is that it''s due to my fiddling with the domain 0 > memory map (I now free pseudophys range 0-1MB to the linux memoy > allocator, which we didn''''t previously do). This might mean that we''re > now allocating memory that gets scribbled on.... > > It might be worth explicitly unmapping the low pseudophys 1MB, not free > it to the memory allocator, and then see if you can boot dom0. Then we > can see if we have any code paths that go straight at low memory, which > is a bug on xenlinux.I wasn''t able to repro these SMP dom0 problems myself. I tried unmapping the low 1MB of pseudophys memory, and got a benign crash in pci_find_bios() (now fixed). It may be that you have some driver that is trying to tamper with ''ISA space''. It''s worth trying to apply the attached patch and see if you can still boot SMP dom0 without crashing. If anyone else is having similar problems then it would be great if they can try this patch too (it''s only good for 32-bit though -- no effect on x86_64). -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir, With your patch on top of 6461:936be0ae823fd3a94ea06d92fdacad65b85d6014 I am still seeing same SMP dom0 boot issue in 8 way SMP box. I am able to reproduce it consistently, I am using the attached config for dom0 kernel. What config are you using? Thanks & Regards, Nitin ------------------------------------------------------------------------ ----------- Sr Software Engineer Open Source Technology Center, Intel Corp -----Original Message----- From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] Sent: Saturday, August 27, 2005 2:17 AM To: Keir Fraser Cc: Kamble, Nitin A; xen-devel Subject: Re: [Xen-devel] Re: SMP dom0 with 8 cpus of i386> One possibility is that it''s due to my fiddling with the domain 0 > memory map (I now free pseudophys range 0-1MB to the linux memoy > allocator, which we didn''''t previously do). This might mean that we''re> now allocating memory that gets scribbled on.... > > It might be worth explicitly unmapping the low pseudophys 1MB, notfree> it to the memory allocator, and then see if you can boot dom0. Then we> can see if we have any code paths that go straight at low memory,which> is a bug on xenlinux.I wasn''t able to repro these SMP dom0 problems myself. I tried unmapping the low 1MB of pseudophys memory, and got a benign crash in pci_find_bios() (now fixed). It may be that you have some driver that is trying to tamper with ''ISA space''. It''s worth trying to apply the attached patch and see if you can still boot SMP dom0 without crashing. If anyone else is having similar problems then it would be great if they can try this patch too (it''s only good for 32-bit though -- no effect on x86_64). -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Keir, > With your patch on top of > 6461:936be0ae823fd3a94ea06d92fdacad65b85d6014 I am still > seeing same SMP dom0 boot issue in 8 way SMP box. > I am able to reproduce it consistently, I am using the > attached config for dom0 kernel. What config are you using?What have you changed relative to the -xen0 kernel config? please post a diff. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Ian,> What have you changed relative to the -xen0 kernel config? please posta> diff.Here is the diff: --- clean/linux-2.6.12-xen0/.config 2005-08-29 15:11:50.000000000 -0700 +++ chgd/linux-2.6.12-xen0/.config 2005-08-29 15:08:54.000000000 -0700 @@ -1,7 +1,7 @@ # # Automatically generated make config: don''t edit # Linux kernel version: 2.6.12-xen0 -# Mon Aug 29 15:11:50 2005 +# Mon Aug 29 15:08:54 2005 # CONFIG_XEN=y CONFIG_ARCH_XEN=y @@ -50,6 +50,7 @@ CONFIG_HOTPLUG=y CONFIG_KOBJECT_UEVENT=y # CONFIG_IKCONFIG is not set +# CONFIG_CPUSETS is not set # CONFIG_EMBEDDED is not set CONFIG_KALLSYMS=y # CONFIG_KALLSYMS_ALL is not set @@ -77,6 +78,7 @@ # CONFIG_MODVERSIONS is not set # CONFIG_MODULE_SRCVERSION_ALL is not set CONFIG_KMOD=y +CONFIG_STOP_MACHINE=y # # X86 Processor Configuration @@ -123,7 +125,10 @@ CONFIG_X86_USE_PPRO_CHECKSUM=y # CONFIG_HPET_TIMER is not set # CONFIG_HPET_EMULATE_RTC is not set -# CONFIG_SMP is not set +CONFIG_SMP=y +# CONFIG_SMP_ALTERNATIVES is not set +CONFIG_NR_CPUS=8 +# CONFIG_SCHED_SMT is not set CONFIG_PREEMPT=y CONFIG_PREEMPT_BKL=y # CONFIG_X86_REBOOTFIXUPS is not set @@ -144,12 +149,11 @@ # CONFIG_REGPARM is not set CONFIG_X86_LOCAL_APIC=y CONFIG_X86_IO_APIC=y +# CONFIG_HOTPLUG_CPU is not set # # Bus options (PCI, PCMCIA, EISA, MCA, ISA) # -CONFIG_X86_UP_APIC=y -CONFIG_X86_UP_IOAPIC=y CONFIG_PCI=y # CONFIG_PCI_GOMMCONFIG is not set # CONFIG_PCI_GODIRECT is not set @@ -178,7 +182,9 @@ # CONFIG_HOTPLUG_PCI is not set CONFIG_GENERIC_HARDIRQS=y CONFIG_GENERIC_IRQ_PROBE=y +CONFIG_X86_SMP=y CONFIG_X86_BIOS_REBOOT=y +CONFIG_X86_TRAMPOLINE=y CONFIG_PC=y CONFIG_SECCOMP=y CONFIG_EARLY_PRINTK=y Thanks & Regards, Nitin ------------------------------------------------------------------------ ----------- Sr Software Engineer Open Source Technology Center, Intel Corp _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 29 Aug 2005, at 19:14, Kamble, Nitin A wrote:> I am able to reproduce it consistently, I am using the attached > config for dom0 kernel. What config are you using?Default but with smp enabled. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Default but with smp enabled.Same here. I am seeing the issue inconsistently on a 4 way box. 8 way system does not have any issue with maxcpus=1. with 8 cpus it is consistent. More no of cpus are causing some corruption. It is always happening at the time of reading/writing the pci mmconfig space. I am debugging here. Thanks & Regards, Nitin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel