Kevin.Buckley@ecs.vuw.ac.nz
2010-Dec-14 01:37 UTC
Re: [Xen-devel] Compile pvops kernel xen/stable-2.6.32.x error
In reference to http://lists.xensource.com/archives/html/xen-devel/2010-10/msg01374.html> arch/x86/xen/mmu.c:163: error: __pcpu_scope_xen_cr3 causes a sectiontype conflict> arch/x86/xen/mmu.c:164: error: __pcpu_scope_xen_current_cr3 causes asection type conflict> arch/x86/xen/mmu.c:163: error: __pcpu_unique_xen_cr3 causes a sectiontype conflict> arch/x86/xen/mmu.c:164: error: __pcpu_unique_xen_current_cr3 causes asection type conflict Firstly, I appreciate that the original thread started a couple of months back but just to say that I recently have seen those messages and discovered what was causing them in the environment I was compiling in so, having found the thread, I thought I''d post my findings, especially as the thread issue doesn''t seem to have been concluded. I first saw the messages when trying to build a kernel which I had configured, starting from a make mrproper; make allmodconfig on a kernel tree pulled as follows: git checkout -b xen/stable-2.6.32.x origin/xen/stable-2.6.32.x as described in: http://bderzhavets.wordpress.com/2010/04/24/set-up-ubuntu-10-04-server-pv-domu-at-xen-4-0-dom0-pvops-2-6-32-10-kernel-dom0-on-top-of-ubuntu-10-04-server/ and using Ubuntu''s GCC 4.4.3 (so a more "recent" one than the OP''s 4.1) I had not seen those messages before re-configuring the kernel and had, indeed built a working, in terms of Xen, kernel using that checked-out source and the instructions above. I then came to look at something else that didn''t appear to work properly for me and so thought I''d go back and modularise as much as possible and so try and see what was being accessed hence the make allmodconfig I did however set all the Xen-related config settings to match those from the working .config file I already had. I thus assumed that there''s some mismatch between compilation in the default config that one gets after doing the: make allmodconfig and the "default" config to which one merely makes Xen-specific changes when doing a make menuconfig and went looking. OK, that''s the background, the info I want to pass on is this: for me, the error messages are effectively toggled if one either defines (see errors) or does not define (no errors) the kernel config value: CONFIG_DEBUG_FORCE_WEAK_PER_CPU which is an option living in lib/Kconfig.debug however the Kconfig help implies it is possibly related to s390 and alpha architectures. That seems to be the only change needed to, and indeed the only change I have found so far which does, make the build fail in arch/x86/xen/mmu.o in the way the OP describes. In case there''ve been any explicit fixes, not posted into this thread, in the meantime, I would have done the source pull around Nov 23rd. Hoping this is useful, apologies if not, Kevin -- Kevin M. Buckley Room: CO327 School of Engineering and Phone: +64 4 463 5971 Computer Science Victoria University of Wellington New Zealand _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Teck Choon Giam
2010-Dec-15 20:55 UTC
Re: [Xen-devel] Compile pvops kernel xen/stable-2.6.32.x error
Hi Kevin, Thanks for sharing... ...> for me, the error messages are effectively toggled if one either > defines (see errors) or does not define (no errors) the kernel > config value: > > CONFIG_DEBUG_FORCE_WEAK_PER_CPUWhat do you mean either define or not define? Do you mean when the above option is enabled or set to y the compilation will fail and errors are what I posted previously in that thread? If that is so, I will have doubt... ... see below... ... # cd /boot # grep CONFIG_DEBUG_FORCE_WEAK_PER_CPU ./config-2.6.32* ./config-2.6.32.25:# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set ./config-2.6.32.25-xenpvops.choon.centos5:# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set ./config-2.6.32.25-xenpvops.choon.centos6:# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set ./config-2.6.32.25-xenU.pvops.centos6.0:# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set ./config-2.6.32.25-xenU.pvops.choon.centos6.0:CONFIG_DEBUG_FORCE_WEAK_PER_CPU=y ./config-2.6.32-44.2.el6.x86_64:CONFIG_DEBUG_FORCE_WEAK_PER_CPU=y config-2.6.32-44.2.el6.x86_64 is stock RHEL6 kernel which I used to build xen pvops using the xen.git tree and got those errors as described in that thread. However, my test compile using config-2.6.32.25-xenU.pvops.choon.centos6.0 is successful and as you can see is with the CONFIG_DEBUG_FORCE_WEAK_PER_CPU=y. However it can''t be booted as I will get kernel panic in xenU/guest. So all my future pvops kernel configuration will not have that option set since I disable majority of the debug options. Xen Minimal OS! start_info: 0xb9c000(VA) nr_pages: 0x40000 shared_inf: 0xde3a2000(MA) pt_base: 0xb9f000(VA) nr_pt_frames: 0xb mfn_list: 0x99c000(VA) mod_start: 0x0(VA) mod_len: 0 flags: 0x0 cmd_line: (hd0)/boot/grub/grub.conf stack: 0x95b840-0x97b840 MM: Init _text: 0x0(VA) _etext: 0x69c3d(VA) _erodata: 0x84000(VA) _edata: 0x8cae0(VA) stack start: 0x95b840(VA) _end: 0x99be48(VA) start_pfn: bad max_pfn: 40000 Mapping memory range 0x1000000 - 0x40000000 setting 0x0-0x84000 readonly skipped 0x1000 MM: Initialise page allocator for da5000(da5000)-40000000(40000000) MM: done Demand map pfns at 40001000-2040001000. Heap resides at 2040002000-4040002000. Initialising timer interface Booting ''Red Hat Enterprise Linux Server (2.6.32.25-xenU.pvops.choon.centos6. 0)'' root (hd0) Error ENOENT when reading the backend path device/vkbd/0/backend Thread "kbdfront" exited. Filesystem type is ext2fs, using whole disk kernel /boot/vmlinuz-2.6.32.25-xenU.pvops.choon.centos6.0 ro root=/dev/xvda1 pa nic=5 panic_timeout=5 selinux=0 LANG=en_US.utf8 initrd /boot/initramfs-2.6.32.25-xenU.pvops.choon.centos6.0.img close blk: backend=/local/domain/0/backend/vbd/12/51713 node=device/vbd/51713 close blk: backend=/local/domain/0/backend/vbd/12/51714 node=device/vbd/51714 Initializing cgroup subsys cpuset Initializing cgroup subsys cpu Linux version 2.6.32.25-xenU.pvops.choon.centos6.0 (root@builder6.choon.net) (gcc version 4.4.4 20100630 (Red Hat 4.4.4-10) (GCC) ) #1 SMP Tue Nov 9 05:01:24 SGT 2010 Command line: ro root=/dev/xvda1 panic=5 panic_timeout=5 selinux=0 LANG=en_US.utf8 KERNEL supported cpus: Intel GenuineIntel AMD AuthenticAMD Centaur CentaurHauls ACPI in unprivileged domain disabled BIOS-provided physical RAM map: Xen: 0000000000000000 - 00000000000a0000 (usable) Xen: 00000000000a0000 - 0000000000100000 (reserved) Xen: 0000000000100000 - 0000000040000000 (usable) DMI not present or invalid. last_pfn = 0x40000 max_arch_pfn = 0x400000000 init_memory_mapping: 0000000000000000-0000000040000000 RAMDISK: 01c7e000 - 14405000 No NUMA configuration found Faking a node at 0000000000000000-0000000040000000 Bootmem setup node 0 0000000000000000-0000000040000000 NODE_DATA [0000000000008000 - 000000000003bfff] bootmap [000000000003c000 - 0000000000043fff] pages 8 (7 early reservations) ==> bootmem [0000000000 - 0040000000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] #1 [0014608000 - 00146af000] XEN PAGETABLES ==> [0014608000 - 00146af000] #2 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000] #3 [0001000000 - 0001c5d418] TEXT DATA BSS ==> [0001000000 - 0001c5d418] #4 [0001c7e000 - 0014405000] RAMDISK ==> [0001c7e000 - 0014405000] #5 [0014405000 - 0014608000] XEN START INFO ==> [0014405000 - 0014608000] #6 [0000100000 - 0000258000] PGTABLE ==> [0000100000 - 0000258000] Zone PFN ranges: DMA 0x00000000 -> 0x00001000 DMA32 0x00001000 -> 0x00100000 Normal 0x00100000 -> 0x00100000 Movable zone start PFN for each node early_node_map[2] active PFN ranges 0: 0x00000000 -> 0x000000a0 0: 0x00000100 -> 0x00040000 SFI: Simple Firmware Interface v0.7 http://simplefirmware.org SMP: Allowing 4 CPUs, 0 hotplug CPUs No local APIC present APIC: disable apic facility PM: Registered nosave memory: 00000000000a0000 - 0000000000100000 Allocating PCI resources starting at 40000000 (gap: 40000000:c0000000) Booting paravirtualized kernel on Xen Xen version: 4.0.2-rc1-pre (preserve-AD) NR_CPUS:4096 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 PERCPU: Embedded 30 pages/cpu @ffff8800146e5000 s92824 r8192 d21864 u122880 pcpu-alloc: s92824 r8192 d21864 u122880 alloc=30*4096 pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 Xen: using vcpu_info placement Built 1 zonelists in Node order, mobility grouping on. Total pages: 258117 Policy zone: DMA32 Kernel command line: ro root=/dev/xvda1 panic=5 panic_timeout=5 selinux=0 LANG=en_US.utf8 PID hash table entries: 4096 (order: 3, 32768 bytes) Initializing CPU#0 Checking aperture... No AGP bridge found Memory: 713456k/1048576k available (4813k kernel code, 384k absent, 334736k reserved, 3867k data, 1136k init) Hierarchical RCU implementation. NR_IRQS:33024 nr_irqs:304 Console: colour dummy device 80x25 console [tty0] enabled console [hvc0] enabled allocated 10485760 bytes of page_cgroup please try ''cgroup_disable=memory'' option if you don''t want memory cgroups installing Xen timer for CPU 0 Detected 2133.446 MHz processor. Calibrating delay loop (skipped), value calculated using timer frequency.. 4266.89 BogoMIPS (lpj=2133446) Security Framework initialized SELinux: Disabled at boot. Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes) Inode-cache hash table entries: 65536 (order: 7, 524288 bytes) Mount-cache hash table entries: 256 Initializing cgroup subsys ns Initializing cgroup subsys cpuacct Initializing cgroup subsys memory Initializing cgroup subsys devices Initializing cgroup subsys freezer Initializing cgroup subsys net_cls CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 4096K CPU 0/0x1 -> Node 0 CPU: Physical Processor ID: 0 CPU: Processor Core ID: 1 Performance Events: unsupported p6 CPU model 15 no PMU driver, software events only. SMP alternatives: switching to UP code ftrace: converting mcount calls to 0f 1f 44 00 00 ftrace: allocating 20529 entries in 81 pages BUG: unable to handle kernel NULL pointer dereference at (null) IP: [<ffffffff814a29d4>] set_cpu_sibling_map+0x12e/0x324 PGD 0 Oops: 0002 [#1] SMP last sysfs file: CPU 0 Modules linked in: Pid: 1, comm: swapper Not tainted 2.6.32.25-xenU.pvops.choon.centos6.0 #1 RIP: e030:[<ffffffff814a29d4>] [<ffffffff814a29d4>] set_cpu_sibling_map+0x12e/0x324 RSP: e02b:ffff88003ea73e80 EFLAGS: 00010246 RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 RDX: ffff8800146e5000 RSI: ffffffff818693c0 RDI: 0000000000000000 RBP: ffff88003ea73ee0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800146faf80 R13: 0000000000000000 R14: 0000000000015f80 R15: 00000020404046e0 FS: 0000000000000000(0000) GS:ffff8800146e5000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000000 CR3: 0000000001001000 CR4: 0000000000002660 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process swapper (pid: 1, threadinfo ffff88003ea72000, task ffff88003ea71460) Stack: ffffffff8100f8cf ffffffff814ac096 ffff88003ea73f10 ffffffff81063fc6 <0> ffff88003ea71460 ffff88003ea70a50 0000000000000000 ffffffff81987ee0 <0> 0000000000001000 0000000000000000 ffffffffffffffff 00000020404046e0 Call Trace: [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1 [<ffffffff814ac096>] ? _spin_unlock_irqrestore+0x16/0x20 [<ffffffff81063fc6>] ? set_cpus_allowed_ptr+0xb6/0x1d0 [<ffffffff8189800a>] xen_smp_prepare_cpus+0x36/0x135 [<ffffffff81893692>] kernel_init+0xc0/0x204 [<ffffffff8101414a>] child_rip+0xa/0x20 [<ffffffff81013311>] ? int_ret_from_sys_call+0x7/0x1b [<ffffffff81013a9d>] ? retint_restore_args+0x5/0x6 [<ffffffff81014140>] ? child_rip+0x0/0x20 Code: c0 01 48 63 d0 e8 0d cb d8 ff 3b 05 07 68 3c 00 0f 8c 54 ff ff ff eb 17 48 8b 14 d5 80 96 86 81 48 c7 c0 b8 e3 00 00 48 8b 04 10 <3e> 0f ab 38 49 8b 84 24 a0 00 00 00 3e 0f ab 18 48 c7 c0 80 5f RIP [<ffffffff814a29d4>] set_cpu_sibling_map+0x12e/0x324 RSP <ffff88003ea73e80> CR2: 0000000000000000 ---[ end trace a7919e7f17c0a725 ]--- Kernel panic - not syncing: Attempted to kill init! Pid: 1, comm: swapper Tainted: G D 2.6.32.25-xenU.pvops.choon.centos6.0 #1 Call Trace: [<ffffffff814a91e0>] panic+0x78/0x137 [<ffffffff8106a0d4>] ? put_files_struct+0xc4/0xf0 [<ffffffff81074c02>] ? exit_ptrace+0xb2/0x140 [<ffffffff8106c66b>] do_exit+0x6bb/0x7e0 [<ffffffff814ad1ab>] oops_end+0xab/0xf0 [<ffffffff81041bf2>] no_context+0xf2/0x260 [<ffffffff81041e85>] __bad_area_nosemaphore+0x125/0x1e0 [<ffffffff81041f53>] bad_area_nosemaphore+0x13/0x20 [<ffffffff814aec41>] do_page_fault+0x2d1/0x380 [<ffffffff814ac535>] page_fault+0x25/0x30 [<ffffffff814a29d4>] ? set_cpu_sibling_map+0x12e/0x324 [<ffffffff8100f8cf>] ? xen_restore_fl_direct_end+0x0/0x1 [<ffffffff814ac096>] ? _spin_unlock_irqrestore+0x16/0x20 [<ffffffff81063fc6>] ? set_cpus_allowed_ptr+0xb6/0x1d0 [<ffffffff8189800a>] xen_smp_prepare_cpus+0x36/0x135 [<ffffffff81893692>] kernel_init+0xc0/0x204 [<ffffffff8101414a>] child_rip+0xa/0x20 [<ffffffff81013311>] ? int_ret_from_sys_call+0x7/0x1b [<ffffffff81013a9d>] ? retint_restore_args+0x5/0x6 [<ffffffff81014140>] ? child_rip+0x0/0x20 Rebooting in 5 seconds.. Since after that compile and test boot failure, I optimized even more for the configuration... ... I have seen Jeremy posted related to this (especially what I submitted to him) to LKML but can''t remember the details. Once again, many thanks for sharing. Now at least I know what is mostly causing that kind of errors and make note to avoid having that option set. Kindest regards, Giam Teck Choon _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Kevin.Buckley@ecs.vuw.ac.nz
2010-Dec-15 21:15 UTC
Re: [Xen-devel] Compile pvops kernel xen/stable-2.6.32.x error
> Hi Kevin, > > Thanks for sharing... ... > > > for me, the error messages are effectively toggled if one either > > defines (see errors) or does not define (no errors) the kernel > > config value: > > > > CONFIG_DEBUG_FORCE_WEAK_PER_CPU > > What do you mean either define or not define? Do you mean when the above > option is enabled or set to y the compilation will fail and errors are > what I posted previously in that thread? If that is so, I will have > doubt...Yep, that''s what I saw. If I have that option checked I get the compilation errors you detailed. If the only change I make to the failing config is to uncheck that option, and then recompile, I don''t. -- Kevin M. Buckley Room: CO327 School of Engineering and Phone: +64 4 463 5971 Computer Science Victoria University of Wellington New Zealand _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Teck Choon Giam
2010-Dec-15 21:24 UTC
Re: [Xen-devel] Compile pvops kernel xen/stable-2.6.32.x error
On Thu, Dec 16, 2010 at 5:15 AM, <Kevin.Buckley@ecs.vuw.ac.nz> wrote:> > Hi Kevin, > > > > Thanks for sharing... ... > > > > > for me, the error messages are effectively toggled if one either > > > defines (see errors) or does not define (no errors) the kernel > > > config value: > > > > > > CONFIG_DEBUG_FORCE_WEAK_PER_CPU > > > > What do you mean either define or not define? Do you mean when the above > > option is enabled or set to y the compilation will fail and errors are > > what I posted previously in that thread? If that is so, I will have > > doubt... > > Yep, that''s what I saw. > > If I have that option checked I get the compilation errors you detailed. > > If the only change I make to the failing config is to uncheck that > option, and then recompile, I don''t. >Thanks for your confirmation ;) Good to know what to avoid when playing with custom kernel compilation... ... Kindest regards, Giam Teck Choon _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel