Hello, I''m trying xen on a dual core Opteron 265 system, but it won''t boot. I''ve tried source installs, and binary installs of both 3.0.2 and 3.0.3. All are the same. When it tries to bring the CPU''s online it reboots the machine. (XEN) Processor #0 15:1 APIC version 15 (XEN) ACPI: LAPIC (acpi_id[0x00] (XEN) Processor #1 15:1 APIC version 15 (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs (XEN) Using ACPI (MADT) for SMP configuration (XEN) Initialising CPU#0 (XEN) Detected 1800.353 Mhz Processor (XEN) CPU0: AMD Flush filter disabled (XEN) CPU: L1 cache 64k (64byes/line), 3 cache (XEN) CPU: L2 cache 1024k (64 bytes/line) (XEN) CPU 0(2) -> Core 0 (XEN) Intel machine check architecture supported (XEN) Intel machine check reporting enabled on CPU#0 (XEN) CPU0: AMD Dual Core Opteron(tm) Processor 265 stepping 02 (XEN) Booting processor 1/1 sip 90000 (XEN) Initialising CPU#1 Then the machine reboots. Motherboard: Tyan K8WE S2877 This is a dual processor, dual core board, but I only have one Opteron265 installed. The machine boots fine (and can see both cores) on a non xen 2.6.16.29kernel. Have I reached a machine check exception? Any ideas on what to customize in the xen kernel to get it to boot? -- John _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Sergey Bondari
2006-Nov-07 12:13 UTC
Re: [Xen-users] Problem initializing dual-core Opteron
Dear John, Tuesday, November 7, 2006, 12:59:53 PM, you wrote: Try switching off APIC tables in your bios setup. This will force kernel not to use SMP. At least you will boot :) In general I would recommend you compiling Dom0 kernel from sources carefully choosing kernel configuration. JH> Hello, JH> I''m trying xen on a dual core Opteron 265 system, but it won''t boot. JH> I''ve tried source installs, and binary installs of both 3.0.2 and 3.0.3. All JH> are the JH> same. When it tries to bring the CPU''s online it reboots the machine. JH> (XEN) Processor #0 15:1 APIC version 15 JH> (XEN) ACPI: LAPIC (acpi_id[0x00] JH> (XEN) Processor #1 15:1 APIC version 15 JH> (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs JH> (XEN) Using ACPI (MADT) for SMP configuration JH> (XEN) Initialising CPU#0 JH> (XEN) Detected 1800.353 Mhz Processor JH> (XEN) CPU0: AMD Flush filter disabled JH> (XEN) CPU: L1 cache 64k (64byes/line), 3 cache JH> (XEN) CPU: L2 cache 1024k (64 bytes/line) (XEN) CPU 0(2) ->> Core 0 JH> (XEN) Intel machine check architecture supported JH> (XEN) Intel machine check reporting enabled on CPU#0 JH> (XEN) CPU0: AMD Dual Core Opteron(tm) Processor 265 stepping 02 JH> (XEN) Booting processor 1/1 sip 90000 JH> (XEN) Initialising CPU#1 JH> Then the machine reboots. JH> Motherboard: Tyan K8WE S2877 JH> This is a dual processor, dual core board, but I only have one Opteron265 JH> installed. JH> The machine boots fine (and can see both cores) on a non xen 2.6.16.29kernel. JH> Have I reached a machine check exception? JH> Any ideas on what to customize in the xen kernel to get it to boot? -- Best regards, Sergey _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
John Hannfield
2006-Nov-07 13:38 UTC
Re: [Xen-users] Problem initializing dual-core Opteron
Thanks Sergey, I switched the BIOS from APIC to PIC, and added ''nolapic nosmp'' to the kernel line and now it boots OK. However I would like to use smp support to get my dual core''s working. I will try again a custom kernel dom0 build. I notice when doing a custom build, xen patches the kernel to offer a "Subarchitecture Type" set to default ''Xen Compatible''. I can imagine you would use this setting for domU kernel, but for dom0 should it not be set to PC Compatible and family set to Opteron? Eg: Subarchitecture Type (PC-compatible) Processor family (Opteron/Athlon64/Hammer/K8) ---> Thanks John _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Petersson, Mats
2006-Nov-07 13:47 UTC
RE: [Xen-users] Problem initializing dual-core Opteron
________________________________ From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of John Hannfield Sent: 07 November 2006 13:38 To: Sergey Bondari Subject: Re: [Xen-users] Problem initializing dual-core Opteron Thanks Sergey, I switched the BIOS from APIC to PIC, and added ''nolapic nosmp'' to the kernel line and now it boots OK. However I would like to use smp support to get my dual core''s working. I will try again a custom kernel dom0 build. I notice when doing a custom build, xen patches the kernel to offer a "Subarchitecture Type" set to default ''Xen Compatible''. I can imagine you would use this setting for domU kernel, but for dom0 should it not be set to PC Compatible and family set to Opteron? To build a Xen kernel you MUST set it to Xen Compatible subarch, otherwise it will not build a Xen kernel - Dom0 and DomU must be this sub-architecture. I know for sure that Xen works on multiprocessor setups, as I''m running one dual core FX-62 and another that is using 2x Opteron 2218. Since the code in Xen at the point of starting processors is completely ignorant about the AMD-V features, it shouldn''t make any difference that you have older Socket 940 type of processor, it still should boot nicely with the standard Xen source code. But of course, it doesn''t work, so something is probably wrong - not sure what or why, tho''. -- Mats Eg: Subarchitecture Type (PC-compatible) Processor family (Opteron/Athlon64/Hammer/K8) ---> Thanks John _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I have had same problem on Intel system - I was not able to boot system due to APIC ACPI problems After I switched the whole dom0 sytem to 64bit (with the same kernel and xen 3.0.2) - it works like a charm. Br Peter _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Petersson, Mats
2006-Nov-07 14:06 UTC
RE: [Xen-users] Problem initializing dual-core Opteron
Interesting, but I doubt this is REALLY the cause of the problem - but there are of course some things that are done differently in 64-bit than 32-bit, mostly because 64-bit "knows" that only certain modern types of hardware supports the 64-bit features... -- Mats ________________________________ From: Peter Braun [mailto:xenware@gmail.com] Sent: 07 November 2006 14:02 To: Petersson, Mats Cc: John Hannfield; Sergey Bondari Subject: Re: [Xen-users] Problem initializing dual-core Opteron I have had same problem on Intel system - I was not able to boot system due to APIC ACPI problems After I switched the whole dom0 sytem to 64bit (with the same kernel and xen 3.0.2) - it works like a charm. Br Peter _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Sergey Bondari
2006-Nov-07 14:21 UTC
Re[2]: [Xen-users] Problem initializing dual-core Opteron
Dear John, Tuesday, November 7, 2006, 2:38:11 PM, you wrote: Do not overcomplicate this stuff around compiling xen0 kernel. Just run make linux-2.6-xen0-config in your xen source folder and choose subarch type XEN, cpu type Opteron and co. Then choose SMP, try playing with big kernel lock (with and without). Do not choose HT at processor setup (its only intel, do not mix it with AMD hypertransport) Than choose carefully your HW (compile it in, do not use modules so you don''t need to play with ramdisks). Do not choose any chips you dont have in your HW. Good luck ;) JH> I switched the BIOS from APIC to PIC, and added ''nolapic nosmp'' to the JH> kernel JH> line and now it boots OK. However I would like to use smp support to get my JH> dual JH> core''s working. JH> I will try again a custom kernel dom0 build. JH> I notice when doing a custom build, xen patches the kernel to JH> offer a "Subarchitecture Type" set to default ''Xen Compatible''. I can JH> imagine you JH> would use this setting for domU kernel, but for dom0 should it not be set JH> to PC Compatible and family set to Opteron? JH> Eg: JH> Subarchitecture Type (PC-compatible) JH> Processor family (Opteron/Athlon64/Hammer/K8) ---> JH> Thanks JH> John -- Best regards, Sergey _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
John Hannfield
2006-Nov-07 14:47 UTC
Re: Re[2]: [Xen-users] Problem initializing dual-core Opteron
OK thanks Sergey, everyone. I am doing a new kernel dom0 compile now, with smp enabled and and with&without CONFIG_PREEMPT_BKL. I do use 32bit debian sarge as my dom0. I could try 64bit debian-amd64 but would like to get 32bit working. On 11/7/06, Sergey Bondari <bondari@itbs.cz> wrote:> > Dear John, > > Tuesday, November 7, 2006, 2:38:11 PM, you wrote: > > Do not overcomplicate this stuff around compiling xen0 kernel. > > Just run make linux-2.6-xen0-config in your xen source folder and > choose subarch type XEN, cpu type Opteron and co. Then choose SMP, try > playing with big kernel lock (with and without). Do not choose HT at > processor setup (its only intel, do not mix it with AMD hypertransport) > > Than choose carefully your HW (compile it in, do not use modules so > you don''t need to play with ramdisks). Do not choose any chips you > dont have in your HW. Good luck ;) > > > JH> I switched the BIOS from APIC to PIC, and added ''nolapic nosmp'' to the > JH> kernel > JH> line and now it boots OK. However I would like to use smp support to > get my > JH> dual > JH> core''s working. > > JH> I will try again a custom kernel dom0 build. > JH> I notice when doing a custom build, xen patches the kernel to > JH> offer a "Subarchitecture Type" set to default ''Xen Compatible''. I can > JH> imagine you > JH> would use this setting for domU kernel, but for dom0 should it not be > set > JH> to PC Compatible and family set to Opteron? > > JH> Eg: > > JH> Subarchitecture Type (PC-compatible) > JH> Processor family (Opteron/Athlon64/Hammer/K8) ---> > > JH> Thanks > JH> John > > > > -- > Best regards, > Sergey > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >-- John _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
John Hannfield
2006-Nov-07 16:00 UTC
Re: Re[2]: [Xen-users] Problem initializing dual-core Opteron
Well I have tried a custom dom0 with cpu type Opteron, subarch set to xen-compatible, and turning PREEMPT_BKL both on and off. It is still the same, it can''t get past activiting the second core of the Opteron 265. My relevant .config sections are shown below. Note, that it *will* boot if I append the ''nolapic'' and ''nosmp'' options to the xen kernel in grub. title Xen 3.0 / XenLinux 2.6 nolapic nosmp noreboot kernel /xen-3.0.gz console=vga nolapic nosmp noreboot module /vmlinuz-2.6-xen root=/dev/hda2 ro console=tty0 But of course I need SMP ! Could this be a BIOS problem? I have my BIOS interrupt set to "APIC" (the other choice is "PIC"). But I do not think it is, as I can boot a non-xen kernel 2.6.16.29 fine with SMP support. # # Automatically generated make config: don''t edit # Linux kernel version: 2.6.16.29-xen # Tue Nov 7 14:06:11 2006 # CONFIG_X86_32=y CONFIG_SEMAPHORE_SLEEPERS=y CONFIG_X86=y CONFIG_MMU=y CONFIG_GENERIC_ISA_DMA=y CONFIG_GENERIC_IOMAP=y CONFIG_ARCH_MAY_HAVE_PC_FDC=y CONFIG_DMI=y # # Code maturity level options # CONFIG_EXPERIMENTAL=y CONFIG_BROKEN_ON_SMP=y CONFIG_INIT_ENV_ARG_LIMIT=32 # # General setup # CONFIG_LOCALVERSION="" # CONFIG_LOCALVERSION_AUTO is not set CONFIG_SWAP=y CONFIG_SYSVIPC=y CONFIG_POSIX_MQUEUE=y CONFIG_BSD_PROCESS_ACCT=y CONFIG_BSD_PROCESS_ACCT_V3=y CONFIG_SYSCTL=y CONFIG_AUDIT=y CONFIG_AUDITSYSCALL=y CONFIG_IKCONFIG=y CONFIG_IKCONFIG_PROC=y CONFIG_INITRAMFS_SOURCE="" CONFIG_UID16=y CONFIG_VM86=y # CONFIG_CC_OPTIMIZE_FOR_SIZE is not set # CONFIG_EMBEDDED is not set CONFIG_KALLSYMS=y # CONFIG_KALLSYMS_ALL is not set # CONFIG_KALLSYMS_EXTRA_PASS is not set CONFIG_HOTPLUG=y CONFIG_PRINTK=y CONFIG_BUG=y CONFIG_ELF_CORE=y CONFIG_BASE_FULL=y CONFIG_FUTEX=y CONFIG_EPOLL=y CONFIG_SHMEM=y CONFIG_CC_ALIGN_FUNCTIONS=0 CONFIG_CC_ALIGN_LABELS=0 CONFIG_CC_ALIGN_LOOPS=0 CONFIG_CC_ALIGN_JUMPS=0 CONFIG_SLAB=y # CONFIG_TINY_SHMEM is not set CONFIG_BASE_SMALL=0 # CONFIG_SLOB is not set CONFIG_OBSOLETE_INTERMODULE=y # # Loadable module support # # CONFIG_MODULES is not set # # Block layer # CONFIG_LBD=y # # IO Schedulers # CONFIG_IOSCHED_NOOP=y CONFIG_IOSCHED_AS=y CONFIG_IOSCHED_DEADLINE=y CONFIG_IOSCHED_CFQ=y CONFIG_DEFAULT_AS=y # CONFIG_DEFAULT_DEADLINE is not set # CONFIG_DEFAULT_CFQ is not set # CONFIG_DEFAULT_NOOP is not set CONFIG_DEFAULT_IOSCHED="anticipatory" # # Processor type and features # # CONFIG_X86_PC is not set CONFIG_X86_XEN=y # CONFIG_X86_ELAN is not set # CONFIG_X86_VOYAGER is not set # CONFIG_X86_NUMAQ is not set # CONFIG_X86_SUMMIT is not set # CONFIG_X86_BIGSMP is not set # CONFIG_X86_VISWS is not set # CONFIG_X86_GENERICARCH is not set # CONFIG_X86_ES7000 is not set # CONFIG_M386 is not set # CONFIG_M486 is not set # CONFIG_M586 is not set # CONFIG_M586TSC is not set # CONFIG_M586MMX is not set # CONFIG_M686 is not set # CONFIG_MPENTIUMII is not set # CONFIG_MPENTIUMIII is not set # CONFIG_MPENTIUMM is not set # CONFIG_MPENTIUM4 is not set # CONFIG_MK6 is not set # CONFIG_MK7 is not set CONFIG_MK8=y # CONFIG_MCRUSOE is not set # CONFIG_MEFFICEON is not set # CONFIG_MWINCHIPC6 is not set # CONFIG_MWINCHIP2 is not set # CONFIG_MWINCHIP3D is not set # CONFIG_MGEODEGX1 is not set # CONFIG_MGEODE_LX is not set # CONFIG_MCYRIXIII is not set # CONFIG_MVIAC3_2 is not set CONFIG_X86_GENERIC=y CONFIG_X86_CMPXCHG=y CONFIG_X86_XADD=y CONFIG_X86_L1_CACHE_SHIFT=7 CONFIG_RWSEM_XCHGADD_ALGORITHM=y CONFIG_GENERIC_CALIBRATE_DELAY=y CONFIG_X86_WP_WORKS_OK=y CONFIG_X86_INVLPG=y CONFIG_X86_BSWAP=y CONFIG_X86_POPAD_OK=y CONFIG_X86_CMPXCHG64=y CONFIG_X86_GOOD_APIC=y CONFIG_X86_INTEL_USERCOPY=y CONFIG_X86_USE_PPRO_CHECKSUM=y CONFIG_X86_TSC=y # CONFIG_SMP is not set # CONFIG_PREEMPT_NONE is not set CONFIG_PREEMPT_VOLUNTARY=y # CONFIG_PREEMPT is not set CONFIG_X86_UP_APIC=y CONFIG_X86_UP_IOAPIC=y CONFIG_X86_LOCAL_APIC=y CONFIG_X86_IO_APIC=y # CONFIG_TOSHIBA is not set # CONFIG_I8K is not set # CONFIG_X86_REBOOTFIXUPS is not set CONFIG_MICROCODE=y CONFIG_X86_CPUID=y CONFIG_SWIOTLB=y _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Sergey Bondari
2006-Nov-07 16:12 UTC
Re[4]: [Xen-users] Problem initializing dual-core Opteron
Dear John, Tuesday, November 7, 2006, 5:00:54 PM, you wrote: JH> Well I have tried a custom dom0 with cpu type Opteron, subarch set to JH> xen-compatible, JH> and turning PREEMPT_BKL both on and off. It is still the same, it can''t get JH> past JH> activiting the second core of the Opteron 265. My relevant .config sections JH> are shown JH> below. JH> Note, that it *will* boot if I append the ''nolapic'' and ''nosmp'' options to JH> the xen JH> kernel in grub. JH> title Xen 3.0 / XenLinux 2.6 nolapic nosmp noreboot JH> kernel /xen-3.0.gz console=vga nolapic nosmp noreboot JH> module /vmlinuz-2.6-xen root=/dev/hda2 ro console=tty0 JH> But of course I need SMP ! JH> Could this be a BIOS problem? I have my BIOS interrupt set to "APIC" (the JH> other choice JH> is "PIC"). But I do not think it is, as I can boot a non-xen kernel JH> 2.6.16.29 fine with SMP JH> support. JH> # JH> # Automatically generated make config: don''t edit JH> # Linux kernel version: 2.6.16.29-xen JH> # Tue Nov 7 14:06:11 2006 JH> # JH> # CONFIG_SMP is not set JH> # CONFIG_PREEMPT_NONE is not set JH> # CONFIG_PREEMPT is not set I don''t what to check every line but this is definitely wrong. CONFIG_SMP should be set to ''y''. I think you should better contact some linux kernel related mailing list for assistance. -- Best regards, Sergey _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
John Hannfield
2006-Nov-07 18:00 UTC
Re: Re[4]: [Xen-users] Problem initializing dual-core Opteron
JH> # CONFIG_SMP is not set Actually that was wrong. A apologise - I posted the wrong .config. CONFIG_SMP was set to y Amazingly I have got it to work, but by changing the BIOS settings, and not the kernel. In my Tyan S2877 Bios under "Hammer Configuration" there is a setting for MemClk with options for 100 Mhz, 133 Mhz, 166Mhz, and 200Mhz. The system will boot into xen and dom0 if MemClk is set to 100 or 133 Mhz. If it is set to 166Mhz or 200Mhz it will not boot. It fails at that point of activating the the CPU cores. I wonder is this memClk to do with the installed RAM or is it processor specific (like onboard cache?) I am using 4 x 1GB kingston ram 400 Mhz DDR ECC Registered CL3 (3-3-3) DIMM Dual Rank, x8 Product: KVR400D8R3A/1G As a side issue, when this xen kernel boots, it only shows the ram as 3GB. When in non-xen it shows all 4GB. Anybody know what setting this memClk BIOS setting has to do with booting the CPU''s? Why should it work when it is set to a lower value? (The default on the motherboard was 200 Mhz) Cheers, John _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users