Hi All, I''m getting the attached panic in the hypervisor on a Sun 6270 running xen-testing.hg changeset 19913:6063c16aeeaa. I can run xen-3.3.1 from the standard SLES 11 distribution (which says that it''s changeset 18546 but it has many patches). Any ideas? thanks, dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Also setting noapic and acpi=off does not change the panic... thanks, dan On Thu, Mar 11, 2010 at 6:40 PM, Dan Gora <dan.gora@gmail.com> wrote:> Hi All, > > I''m getting the attached panic in the hypervisor on a Sun 6270 running > xen-testing.hg changeset 19913:6063c16aeeaa. I can run xen-3.3.1 from > the standard SLES 11 distribution (which says that it''s changeset > 18546 but it has many patches). Any ideas? > > thanks, > dan_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 11/03/2010 21:40, "Dan Gora" <dan.gora@gmail.com> wrote:> I''m getting the attached panic in the hypervisor on a Sun 6270 running > xen-testing.hg changeset 19913:6063c16aeeaa. I can run xen-3.3.1 from > the standard SLES 11 distribution (which says that it''s changeset > 18546 but it has many patches). Any ideas?The system has quiet a few CPUs which puts Xen into ''bigsmp'' mode, and maybe there is a problem initialising that. Will have to try to reproduce this. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Why it will fail on 0xffff828bfffff0e0 ? I try to build 19913 and seems it should be 0xffff828bffffe0e0. If you can paste your objdump output for init_apic_ldr_phys, maybe it will be helpful. BTW, if SLES 11 works for you, why not continue that? --jyh ffff828c801879d0 <init_apic_ldr_phys>: ........... ffff828c801879f8: b8 ff ff ff ff mov $0xffffffff,%eax ffff828c801879fd: a3 e0 e0 ff ff 8b 82 mov %eax,0xffff828bffffe0e0 ffff828c80187a04: ff ff ffff828c80187a06: a1 d0 e0 ff ff 8b 82 mov 0xffff828bffffe0d0,%eax ffff828c80187a0d: ff ff ffff828c80187a0f: 25 ff ff ff 00 and $0xffffff,%eax ffff828c80187a14: a3 d0 e0 ff ff 8b 82 mov %eax,0xffff828bffffe0d0 ...............>-----Original Message----- >From: xen-devel-bounces@lists.xensource.com >[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Dan Gora >Sent: Friday, March 12, 2010 5:55 AM >To: xen-devel@lists.xensource.com >Subject: [Xen-devel] Re: Panic on boot on Sun Blade 6270 > >Also setting noapic and acpi=off does not change the panic... > >thanks, >dan > >On Thu, Mar 11, 2010 at 6:40 PM, Dan Gora <dan.gora@gmail.com> wrote: >> Hi All, >> >> I''m getting the attached panic in the hypervisor on a Sun 6270 running >> xen-testing.hg changeset 19913:6063c16aeeaa. I can run xen-3.3.1 from >> the standard SLES 11 distribution (which says that it''s changeset >> 18546 but it has many patches). Any ideas? >> >> thanks, >> dan > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Mar 11, 2010 at 10:35 PM, Jiang, Yunhong <yunhong.jiang@intel.com> wrote:> Why it will fail on 0xffff828bfffff0e0 ? I try to build 19913 and seems it should be 0xffff828bffffe0e0. > If you can paste your objdump output for init_apic_ldr_phys, maybe it will be helpful.Here is init_apic_ldr_phys from ''objdump -d xen-syms-3.4.3-rc4-pre'': ffff828c80183d48 <init_apic_ldr_flat>: ffff828c80183d48: 55 push %rbp ffff828c80183d49: 48 89 e5 mov %rsp,%rbp ffff828c80183d4c: 83 3d cd 73 09 00 00 cmpl $0x0,0x973cd(%rip) # ffff828c8021b120 <x2apic_enabled> ffff828c80183d53: 74 4d je ffff828c80183da2 <init_apic_ldr_flat+0x5a> ffff828c80183d55: ba 00 00 00 00 mov $0x0,%edx ffff828c80183d5a: b8 ff ff ff ff mov $0xffffffff,%eax ffff828c80183d5f: b9 0e 08 00 00 mov $0x80e,%ecx ffff828c80183d64: 0f 30 wrmsr ffff828c80183d66: b1 0d mov $0xd,%cl ffff828c80183d68: 0f 32 rdmsr ffff828c80183d6a: 48 89 c2 mov %rax,%rdx ffff828c80183d6d: 81 e2 ff ff ff 00 and $0xffffff,%edx ffff828c80183d73: 48 c7 c0 00 80 ff ff mov $0xffffffffffff8000,%rax ffff828c80183d7a: 48 21 e0 and %rsp,%rax ffff828c80183d7d: 48 0d 28 7f 00 00 or $0x7f28,%rax ffff828c80183d83: 8b 88 c8 00 00 00 mov 0xc8(%rax),%ecx ffff828c80183d89: b8 00 00 00 01 mov $0x1000000,%eax ffff828c80183d8e: 48 d3 e0 shl %cl,%rax ffff828c80183d91: 48 09 d0 or %rdx,%rax ffff828c80183d94: ba 00 00 00 00 mov $0x0,%edx ffff828c80183d99: b9 0d 08 00 00 mov $0x80d,%ecx ffff828c80183d9e: 0f 30 wrmsr ffff828c80183da0: eb 43 jmp ffff828c80183de5 <init_apic_ldr_flat+0x9d> ffff828c80183da2: b8 ff ff ff ff mov $0xffffffff,%eax ffff828c80183da7: a3 e0 f0 ff ff 8b 82 mov %eax,0xffff828bfffff0e0 ffff828c80183dae: ff ff ffff828c80183db0: 48 be d0 f0 ff ff 8b mov $0xffff828bfffff0d0,%rsi ffff828c80183db7: 82 ff ff ffff828c80183dba: 8b 16 mov (%rsi),%edx ffff828c80183dbc: 81 e2 ff ff ff 00 and $0xffffff,%edx ffff828c80183dc2: 48 c7 c0 00 80 ff ff mov $0xffffffffffff8000,%rax ffff828c80183dc9: 48 21 e0 and %rsp,%rax ffff828c80183dcc: 48 0d 28 7f 00 00 or $0x7f28,%rax ffff828c80183dd2: 8b 88 c8 00 00 00 mov 0xc8(%rax),%ecx ffff828c80183dd8: b8 00 00 00 01 mov $0x1000000,%eax ffff828c80183ddd: 48 d3 e0 shl %cl,%rax ffff828c80183de0: 48 09 d0 or %rdx,%rax ffff828c80183de3: 89 06 mov %eax,(%rsi) ffff828c80183de5: c9 leaveq ffff828c80183de6: c3 retq I don''t quite understand why we have such different addresses. I''m thinking maybe I built it wrong or something. I built it with debug set to ''y'' in Config.mk, then just did ''make dist'', copied the resultant ''dist'' directory to the test machine and ran ''sh ./install.sh'' on the target machine, and rebooted with: title Xen-3.4.3-rc1.dan -- SUSE Linux Enterprise Server 11 - 2.6.27.19-5 (xen).dan root (hd0,0) kernel /xen-3.4.3-rc4-pre.gz crashkernel=128M@16M iommu=1 loglvl=all guest_loglvl=all console=vga,com1 com1=auto module /vmlinuz-2.6.27.19-5-xen.dan root=/dev/dm-5 resume=/dev/dm-2 splash=silent showopts console=ttyS0,115200 quiet module /initrd-2.6.27.19-5-xen.dan> BTW, if SLES 11 works for you, why not continue that?ugh.. long story. It works in the sense that it boots and can run VMs. My problem is that I''m trying to get the board that our company makes to work in an HVM guest properly.. *something* is messing with my board''s PCI bar registers and I cannot figure out what it is. I added printks to all the kernel functions which touch the BAR registers and to the pciback driver as well, but neither of these is occurring, so I figured the problem is in qemu, but to test that I need to be able to compile it. I thought that the easiest path there would simply to compile the latest stuff, but it panicked early in the hypervisor so I thought that I''d report it... thanks, dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
The only possibility I can see is changeset 18722 is missed from your build. So you are really clone a CLEAN xen tree and do the build from scratch? --jyh>-----Original Message----- >From: Dan Gora [mailto:dan.gora@gmail.com] >Sent: Friday, March 12, 2010 12:01 PM >To: Jiang, Yunhong >Cc: xen-devel@lists.xensource.com >Subject: Re: [Xen-devel] Re: Panic on boot on Sun Blade 6270 > >On Thu, Mar 11, 2010 at 10:35 PM, Jiang, Yunhong ><yunhong.jiang@intel.com> wrote: >> Why it will fail on 0xffff828bfffff0e0 ? I try to build 19913 and seems it should be >0xffff828bffffe0e0. >> If you can paste your objdump output for init_apic_ldr_phys, maybe it will be >helpful. > >Here is init_apic_ldr_phys from ''objdump -d xen-syms-3.4.3-rc4-pre'': > >ffff828c80183d48 <init_apic_ldr_flat>: >ffff828c80183d48: 55 push %rbp >ffff828c80183d49: 48 89 e5 mov %rsp,%rbp >ffff828c80183d4c: 83 3d cd 73 09 00 00 cmpl >$0x0,0x973cd(%rip) # ffff828c8021b120 <x2apic_enabled> >ffff828c80183d53: 74 4d je >ffff828c80183da2 <init_apic_ldr_flat+0x5a> >ffff828c80183d55: ba 00 00 00 00 mov $0x0,%edx >ffff828c80183d5a: b8 ff ff ff ff mov $0xffffffff,%eax >ffff828c80183d5f: b9 0e 08 00 00 mov $0x80e,%ecx >ffff828c80183d64: 0f 30 wrmsr >ffff828c80183d66: b1 0d mov $0xd,%cl >ffff828c80183d68: 0f 32 rdmsr >ffff828c80183d6a: 48 89 c2 mov %rax,%rdx >ffff828c80183d6d: 81 e2 ff ff ff 00 and $0xffffff,%edx >ffff828c80183d73: 48 c7 c0 00 80 ff ff mov $0xffffffffffff8000,%rax >ffff828c80183d7a: 48 21 e0 and %rsp,%rax >ffff828c80183d7d: 48 0d 28 7f 00 00 or $0x7f28,%rax >ffff828c80183d83: 8b 88 c8 00 00 00 mov 0xc8(%rax),%ecx >ffff828c80183d89: b8 00 00 00 01 mov $0x1000000,%eax >ffff828c80183d8e: 48 d3 e0 shl %cl,%rax >ffff828c80183d91: 48 09 d0 or %rdx,%rax >ffff828c80183d94: ba 00 00 00 00 mov $0x0,%edx >ffff828c80183d99: b9 0d 08 00 00 mov $0x80d,%ecx >ffff828c80183d9e: 0f 30 wrmsr >ffff828c80183da0: eb 43 jmp >ffff828c80183de5 <init_apic_ldr_flat+0x9d> >ffff828c80183da2: b8 ff ff ff ff mov $0xffffffff,%eax >ffff828c80183da7: a3 e0 f0 ff ff 8b 82 mov %eax,0xffff828bfffff0e0 >ffff828c80183dae: ff ff >ffff828c80183db0: 48 be d0 f0 ff ff 8b mov $0xffff828bfffff0d0,%rsi >ffff828c80183db7: 82 ff ff >ffff828c80183dba: 8b 16 mov (%rsi),%edx >ffff828c80183dbc: 81 e2 ff ff ff 00 and $0xffffff,%edx >ffff828c80183dc2: 48 c7 c0 00 80 ff ff mov $0xffffffffffff8000,%rax >ffff828c80183dc9: 48 21 e0 and %rsp,%rax >ffff828c80183dcc: 48 0d 28 7f 00 00 or $0x7f28,%rax >ffff828c80183dd2: 8b 88 c8 00 00 00 mov 0xc8(%rax),%ecx >ffff828c80183dd8: b8 00 00 00 01 mov $0x1000000,%eax >ffff828c80183ddd: 48 d3 e0 shl %cl,%rax >ffff828c80183de0: 48 09 d0 or %rdx,%rax >ffff828c80183de3: 89 06 mov %eax,(%rsi) >ffff828c80183de5: c9 leaveq >ffff828c80183de6: c3 retq > >I don''t quite understand why we have such different addresses. I''m >thinking maybe I built it wrong or something. I built it with debug >set to ''y'' in Config.mk, then just did ''make dist'', copied the >resultant ''dist'' directory to the test machine and ran ''sh >./install.sh'' on the target machine, and rebooted with: > >title Xen-3.4.3-rc1.dan -- SUSE Linux Enterprise Server 11 - >2.6.27.19-5 (xen).dan > root (hd0,0) > kernel /xen-3.4.3-rc4-pre.gz crashkernel=128M@16M iommu=1 >loglvl=all guest_loglvl=all console=vga,com1 com1=auto > module /vmlinuz-2.6.27.19-5-xen.dan root=/dev/dm-5 >resume=/dev/dm-2 splash=silent showopts console=ttyS0,115200 quiet > module /initrd-2.6.27.19-5-xen.dan > > >> BTW, if SLES 11 works for you, why not continue that? > >ugh.. long story. It works in the sense that it boots and can run >VMs. My problem is that I''m trying to get the board that our company >makes to work in an HVM guest properly.. *something* is messing with >my board''s PCI bar registers and I cannot figure out what it is. I >added printks to all the kernel functions which touch the BAR >registers and to the pciback driver as well, but neither of these is >occurring, so I figured the problem is in qemu, but to test that I >need to be able to compile it. I thought that the easiest path there >would simply to compile the latest stuff, but it panicked early in the >hypervisor so I thought that I''d report it... > >thanks, >dan_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 12/03/2010 04:01, "Dan Gora" <dan.gora@gmail.com> wrote:> On Thu, Mar 11, 2010 at 10:35 PM, Jiang, Yunhong > <yunhong.jiang@intel.com> wrote: >> Why it will fail on 0xffff828bfffff0e0 ? I try to build 19913 and seems it >> should be 0xffff828bffffe0e0. >> If you can paste your objdump output for init_apic_ldr_phys, maybe it will be >> helpful. > > Here is init_apic_ldr_phys from ''objdump -d xen-syms-3.4.3-rc4-pre'':Er..> ffff828c80183d48 <init_apic_ldr_flat>:...no it''s not. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Mar 12, 2010 at 3:20 AM, Jiang, Yunhong <yunhong.jiang@intel.com> wrote:> The only possibility I can see is changeset 18722 is missed from your build. > So you are really clone a CLEAN xen tree and do the build from scratch? >yes... I cloned xen-testing-3.4.hg. I''m not proficient enough with hg to remove a single commit. d _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Mar 12, 2010 at 4:13 AM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:>> Here is init_apic_ldr_phys from ''objdump -d xen-syms-3.4.3-rc4-pre'': > > Er.. > >> ffff828c80183d48 <init_apic_ldr_flat>: > > ...no it''s not.ok.. what is it then? I''m pretty darn sure that''s what I typed. d _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I guess that I can try and redo everything again. I''ve been working with xen a grand total of two weeks, so I''m a little confused about the building process. Do you have to somehow build xen against the kernel that you are going to use or can you just use the default 2.6.18 kernel which gets downloaded automatically? Like I said, I just cloned the repo (xen-testing-3.4.hg) and ran ''make dist''. Since it''s panicking so early in the hypervisor I didn''t think that it could possibly be due to some kernel difference. Also, is it expected that the default .hg tree xen will *just work* on a SLES distro by running install.sh or do you need a bunch of SuSE specific patches? thanks dan On Fri, Mar 12, 2010 at 10:37 AM, Dan Gora <dan.gora@gmail.com> wrote:> On Fri, Mar 12, 2010 at 4:13 AM, Keir Fraser <keir.fraser@eu.citrix.com> wrote: >>> Here is init_apic_ldr_phys from ''objdump -d xen-syms-3.4.3-rc4-pre'': >> >> Er.. >> >>> ffff828c80183d48 <init_apic_ldr_flat>: >> >> ...no it''s not. > > ok.. what is it then? I''m pretty darn sure that''s what I typed. > > d >-- Dan Gora Software Engineer Adax, Inc. Av Dona Maria Alves, 793-LJ 04 Ubatuba, SP CEP 11680-000 Brasil Tel: +55 (12) 3833-1021 (Brazil and outside of US) : +1 (510) 859-4801 (Inside of US) <= Note new number! : dan_gora (Skype) email: dg@adax.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Mar 12, 2010 at 10:43:08AM -0300, Dan Gora wrote:> I guess that I can try and redo everything again. I''ve been working > with xen a grand total of two weeks, so I''m a little confused about > the building process. Do you have to somehow build xen against the > kernel that you are going to use or can you just use the default > 2.6.18 kernel which gets downloaded automatically? >Xen hypervisor and tools can be built separately from the dom0 kernel.> Like I said, I > just cloned the repo (xen-testing-3.4.hg) and ran ''make dist''. Since > it''s panicking so early in the hypervisor I didn''t think that it could > possibly be due to some kernel difference. >You could just grab the latest xen-3.4-testing.hg and do: make xen make tools make stubdom Now the compiled xen+tools+stubdom is in dist/ directory. make install-xen make install-tools make install-stubdom After that you could grab whatever dom0 kernel, compile and install it. latest 2.6.18-xen.hg: hg clone http://xenbits.xen.org/linux-2.6.18-xen.hg Or instructions for 2.6.31 / 2.6.32 pv_ops dom0 here: http://wiki.xensource.com/xenwiki/XenParavirtOps> Also, is it expected that > the default .hg tree xen will *just work* on a SLES distro by running > install.sh or do you need a bunch of SuSE specific patches? >The default hg tree should work. -- Pasi> thanks > dan > > > On Fri, Mar 12, 2010 at 10:37 AM, Dan Gora <dan.gora@gmail.com> wrote: > > On Fri, Mar 12, 2010 at 4:13 AM, Keir Fraser <keir.fraser@eu.citrix.com> wrote: > >>> Here is init_apic_ldr_phys from ''objdump -d xen-syms-3.4.3-rc4-pre'': > >> > >> Er.. > >> > >>> ffff828c80183d48 <init_apic_ldr_flat>: > >> > >> ...no it''s not. > > > > ok.. what is it then? I''m pretty darn sure that''s what I typed. > > > > d > > > > > > -- > Dan Gora > Software Engineer > > Adax, Inc. > Av Dona Maria Alves, 793-LJ 04 > Ubatuba, SP > CEP 11680-000 > Brasil > > Tel: +55 (12) 3833-1021 (Brazil and outside of US) > : +1 (510) 859-4801 (Inside of US) <= Note new number! > : dan_gora (Skype) > > email: dg@adax.com > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
So I tried it all over again from scratch with the xen-testing.hg tree at commit 19924 and it doesn''t panic any more. It gets to booting the kernel, then locks up when trying to bring the networking up, but at least it doesn''t have the same panic anymore. Sorry about the false alarm. I probably gaffed something with hg on the previous go round.. thanks dan On Fri, Mar 12, 2010 at 10:43 AM, Dan Gora <dan.gora@gmail.com> wrote:> I guess that I can try and redo everything again. I''ve been working > with xen a grand total of two weeks, so I''m a little confused about > the building process. Do you have to somehow build xen against the > kernel that you are going to use or can you just use the default > 2.6.18 kernel which gets downloaded automatically? Like I said, I > just cloned the repo (xen-testing-3.4.hg) and ran ''make dist''. Since > it''s panicking so early in the hypervisor I didn''t think that it could > possibly be due to some kernel difference. Also, is it expected that > the default .hg tree xen will *just work* on a SLES distro by running > install.sh or do you need a bunch of SuSE specific patches? > > thanks > dan > > > On Fri, Mar 12, 2010 at 10:37 AM, Dan Gora <dan.gora@gmail.com> wrote: >> On Fri, Mar 12, 2010 at 4:13 AM, Keir Fraser <keir.fraser@eu.citrix.com> wrote: >>>> Here is init_apic_ldr_phys from ''objdump -d xen-syms-3.4.3-rc4-pre'': >>> >>> Er.. >>> >>>> ffff828c80183d48 <init_apic_ldr_flat>: >>> >>> ...no it''s not. >> >> ok.. what is it then? I''m pretty darn sure that''s what I typed. >> >> d_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Mar 12, 2010 at 1:44 PM, Pasi Kärkkäinen <pasik@iki.fi> wrote:>> Also, is it expected that >> the default .hg tree xen will *just work* on a SLES distro by running >> install.sh or do you need a bunch of SuSE specific patches? >> > > The default hg tree should work.Thanks for the tips Pasi.. I guess I did pretty much that, but I just did a ''make dist'' and built everything including the 2.6.18 kernel, I moved everything to the test machine and ran ''sh ./install.sh'' and added a grub line to boot the new xen: title Xen-3.4.3-rc1.dan -- SUSE Linux Enterprise Server 11 - 2.6.27.19-5 (xen).dan root (hd0,0) kernel /xen-3.4.3-rc4-pre.gz crashkernel=128M@16M iommu=1 loglvl=all guest_loglvl=all console=vga,com1 com1=auto module /vmlinuz-2.6.27.19-5-xen.dan root=/dev/dm-5 resume=/dev/dm-2 splash=silent showopts console=ttyS0,115200 quiet module /initrd-2.6.27.19-5-xen.dan so it should use the new hypervisor (xen-3.4.3-rc4-pre.gz) but the original SLES 11 kernel /vmlinuz-2.6.27.19-5-xen.dan ( the stock kernel with a couple of printks thrown in). This boots to the kernel, but then it locks up the machine when it tries to bring the networking up: Mounting local file systems... /proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) debugfs on /sys/kernel/debug type debugfs (rw) udev on /dev type tmpfs (rw) devpts on /dev/pts type devpts (rw,mode=0620,gid=5) /dev/mapper/ddf1_disk1_part1 on /boot type ext2 (rw,acl,user_xattr) /dev/mapper/ddf1_disk1_part3 on /temp type ext3 (rw,acl,user_xattr) done Setting current sysctl status from /etc/sysctl.conf done Enabling syn flood protection done Disabling IP forwarding done done Loading fuse module done Mounting fuse control filesystem done Activating remaining swap-devices in /etc/fstab... done Creating /var/log/boot.msg done Setting up linker cache (/etc/ld.so.cache) using ldconfig done ATTENTION: You have modified /etc/resolv.conf. Leaving it untouched... You can find my version in /etc/resolv.conf.netconfig ... ATTENTION: You have modified /etc/yp.conf. Leaving it untouched... You can find my version in /etc/yp.conf.netconfig ... Setting up hostname ''sunblade1'' done Setting up NIS domainname ''adax.co.uk'' done Setting up loopback interface lo lo IP address: 127.0.0.1/8 IP address: 127.0.0.2/8 Bridge firewalling registered After about 5 minutes of sitting there it crashed: BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 IP: [<ffffffff8020e9cb>] profile_pc+0x30/0x3d PGD 0 Oops: 0000 [1] SMP last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:20:05.0/class CPU 0 Modules linked in: bridge(N) stp(N) fuse(N) ext2(N) loop(N) rtc_cmos(N) i2c_i80) Supported: No Pid: 92, comm: kswapd0 Tainted: G 2.6.27.19-5-xen.dan #5 RIP: e030:[<ffffffff8020e9cb>] [<ffffffff8020e9cb>] profile_pc+0x30/0x3d RSP: e02b:ffffffff80783db8 EFLAGS: 00010002 RAX: 0000000000000000 RBX: ffffffff8049077c RCX: ffffffff8067e320 RDX: ffff880081fa5000 RSI: 0000000000000400 RDI: ffffffff8049077c RBP: ffffffff80783dc8 R08: 0000000000000c31 R09: ffff8803d8608878 R10: ffffffff80783d38 R11: 000000000000000c R12: ffff8803d860b838 R13: ffffffff80774020 R14: 00000000006187a6 R15: ffff880002719000 FS: 00007fde48ab66f0(0000) GS:ffffffff80785080(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kswapd0 (pid: 92, threadinfo ffff8803d860a000, task ffff8803d8608840) Stack: ffff8803d860b838 0000000000000001 ffffffff80783de8 ffffffff8024da1a 0000000000000000 0000000000000020 ffffffff80783e98 ffffffff8020e987 ffffffff80490942 ffffffff806b7108 ffffffff807e9700 ffffffff80774000 Call Trace: [<ffffffff8024da1a>] profile_tick+0x5f/0x7d [<ffffffff8020e987>] timer_interrupt+0x436/0x44a [<ffffffff80270594>] handle_IRQ_event+0x50/0x9c [<ffffffff80271757>] handle_percpu_irq+0x45/0x73 [<ffffffff8020d8e3>] do_IRQ+0x4a/0x92 [<ffffffff803dbce8>] evtchn_do_upcall+0x1a9/0x275 [<ffffffff8020bcae>] do_hypervisor_callback+0x1e/0x30 [<ffffffff8049077c>] _read_unlock_irqrestore+0x58/0x59 DWARF2 unwinder stuck at _read_unlock_irqrestore+0x58/0x59 Leftover inexact backtrace: [<ffffffffa012c8a7>] ? dm_get_table+0x35/0x3d [dm_mod] [<ffffffffa012ca10>] ? __split_bio+0x1e/0x47f [dm_mod] [<ffffffff802837ac>] ? __set_page_dirty_nobuffers+0x12/0x18d [<ffffffff802a691d>] ? __mem_cgroup_uncharge_common+0x10/0x163 [<ffffffff802896ae>] ? __dec_zone_state+0x9/0x77 [<ffffffff80490942>] ? _spin_lock_irq+0xc/0x71 [<ffffffff804902f4>] ? __down_read+0x1a/0x120 [<ffffffff8027f2ae>] ? mempool_alloc_slab+0x16/0x18 [<ffffffffa012d056>] ? dm_request+0x13d/0x15f [dm_mod] [<ffffffff8033d1c0>] ? generic_make_request+0x39c/0x3df [<ffffffff802d282a>] ? bvec_alloc_bs+0x86/0xad [<ffffffff802d2780>] ? bio_init+0xd/0x31 [<ffffffff802d28a1>] ? bio_alloc_bioset+0x50/0x97 [<ffffffff804909b9>] ? _spin_lock_irqsave+0x12/0x96 [<ffffffff8033d2ce>] ? submit_bio+0xcb/0xd4 [<ffffffff8029c3ba>] ? swap_writepage+0xf6/0x103 [<ffffffff80287748>] ? shrink_page_list+0x37b/0x5f3 [<ffffffff8021d359>] ? ptep_clear_flush_young+0x7c/0x8b [<ffffffff80286829>] ? __isolate_lru_page+0x9/0x62 [<ffffffff802869e8>] ? isolate_lru_pages+0x166/0x210 [<ffffffff80287b90>] ? shrink_inactive_list+0x1d0/0x51d [<ffffffff80287fe8>] ? shrink_zone+0x10b/0x12e [<ffffffff802885f1>] ? kswapd+0x3f3/0x583 [<ffffffff80286a92>] ? isolate_pages_global+0x0/0x39 [<ffffffff8024860a>] ? autoremove_wake_function+0x0/0x3d [<ffffffff802881fe>] ? kswapd+0x0/0x583 [<ffffffff802480bf>] ? kthread+0x4e/0x7b [<ffffffff8020bef9>] ? child_rip+0xa/0x11 [<ffffffff80248071>] ? kthread+0x0/0x7b [<ffffffff8020beef>] ? child_rip+0x0/0x11 Code: 54 53 e8 89 ca ff ff f6 87 88 00 00 00 03 48 8b 9f 80 00 00 00 49 89 fc 7 RIP [<ffffffff8020e9cb>] profile_pc+0x30/0x3d RSP <ffffffff80783db8> CR2: 0000000000000008 ---[ end trace a2b3d37002a3cae5 ]--- Kernel panic - not syncing: Aiee, killing interrupt handler! ------------[ cut here ]------------ WARNING: at kernel/smp.c:331 smp_call_function_mask+0x5a/0x1e1() Modules linked in: bridge(N) stp(N) fuse(N) ext2(N) loop(N) rtc_cmos(N) i2c_i80) Is there some difference in the network configuration or something that could cause this? thanks dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> Dan Gora <dan.gora@gmail.com> 12.03.10 18:37 >>> >so it should use the new hypervisor (xen-3.4.3-rc4-pre.gz) but the >original SLES 11 kernel /vmlinuz-2.6.27.19-5-xen.dan ( the stock >kernel with a couple of printks thrown in). > >This boots to the kernel, but then it locks up the machine when it >tries to bring the networking up: >... >After about 5 minutes of sitting there it crashed: > >BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 >IP: [<ffffffff8020e9cb>] profile_pc+0x30/0x3dI would have been tempted to analyze this (given that it may point out a general problem with the SLE11 kernel) if this wasn''t a kernel that you built on your own and if this wasn''t a really old one. I would guess that it''s not only a couple of added printk()-s, but also a different .config that you''re using. Namely (not having seen the disassembly of your kernel''s profile_pc()) I could easily imagine this oops to be a result of you building with CONFIG_FRAME_POINTER=y. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, Mar 15, 2010 at 6:21 AM, Jan Beulich <JBeulich@novell.com> wrote:>>After about 5 minutes of sitting there it crashed: >> >>BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 >>IP: [<ffffffff8020e9cb>] profile_pc+0x30/0x3d > > I would have been tempted to analyze this (given that it may point > out a general problem with the SLE11 kernel) if this wasn''t a kernel > that you built on your own and if this wasn''t a really old one. I > would guess that it''s not only a couple of added printk()-s, but also > a different .config that you''re using. Namely (not having seen the > disassembly of your kernel''s profile_pc()) I could easily imagine > this oops to be a result of you building with CONFIG_FRAME_POINTER=y.:)... Yes I did build with CONFIG_FRAME_POINTER=y. I guess I considered the panic more incidental... Really the problem was that the machine locked up on boot. I didn''t realize that CONFIG_FRAME_POINTER could cause a panic.. I thought that it just turned on using the frame pointer register so that backtraces would work. I tried to install the xen 3.4.1 from the SuSE rpms after this and also could not get that to work. That too would panic on boot up. I tried to install the xen 4.0.0 rpms, but quickly ran into prerequisite problems and just gave up and went back to using xen 3.3.1, building it from the SRPMS (I still cannot get anything to work just building it from the xen hg repo). I realize that this Sun box is probably pretty funky (it''s a PCI express module based machine) so it might have some peculiar issues that would be of interest, but I''ve got to figure this other issue out first because I have customers on my back. If you are interested, Jan, I can go back and get the backtrace from the 3.4.1 panic, but if SLES 11 is too old to try and debug I''ll just forget about it.. If there''s time later and there is more recent stuff that we can try and test, let me know what it is and I can give it a go. thanks, dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> Dan Gora <dan.gora@gmail.com> 16.03.10 03:39 >>> >:)... Yes I did build with CONFIG_FRAME_POINTER=y. I guess I >considered the panic more incidental... Really the problem was that >the machine locked up on boot. I didn''t realize that >CONFIG_FRAME_POINTER could cause a panic.. I thought that it just >turned on using the frame pointer register so that backtraces would >work.Our kernels have working backtraces without CONFIG_FRAME_POINTER, which is why we are (happily) disabling this option.>I tried to install the xen 3.4.1 from the SuSE rpms after this and >also could not get that to work. That too would panic on boot up. IIt definitely shouldn''t.>tried to install the xen 4.0.0 rpms, but quickly ran into prerequisite >problems and just gave up and went back to using xen 3.3.1, building >it from the SRPMS (I still cannot get anything to work just building >it from the xen hg repo).So are you saying even plain SLE11 doesn''t work on that box? If so, why didn''t you report this to our support folks yet?>I realize that this Sun box is probably pretty funky (it''s a PCI >express module based machine) so it might have some peculiar issues >that would be of interest, but I''ve got to figure this other issue out >first because I have customers on my back. If you are interested, >Jan, I can go back and get the backtrace from the 3.4.1 panic, but if >SLES 11 is too old to try and debug I''ll just forget about it..As said above - it''s supposed to work (both with the 3.3.1 that SLE11 ships with [or really, with the most recent maintenance updates in place for both kernel and Xen] and upstream versions down to 3.2.0). The only limitation I know about for certain (Sun?) systems is that not all interrupts may be usable by their devices/drivers. That limitation goes away only with 4.0 (i.e. earlier upstream versions won''t - afaict - get you anywhere either if this is the case). Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Mar 16, 2010 at 7:17 AM, Jan Beulich <JBeulich@novell.com> wrote:> So are you saying even plain SLE11 doesn''t work on that box?No, I''m not saying that.. SLES 11 + Xen 3.3.1 "works" (yes I have a problem with it, but I''m still trying to figure it out). In trying to figure out my other problem I wanted to upgrade Xen to something later but was not having any luck building and installing Xen from the hg repo. So I went to download.suse.com and downloaded the Xen 3.4.1 SRPMS, used rpmbuild to build the RPMS, installed those and that panics on boot. I could not install the 4.0.0 RPMs because there were conflicts with the required prerequisites and what I already installed, so I gave up.> If so, why didn''t you report this to our support folks yet?I haven''t reported the 3.4.1 crash just because: 1) I just found it a couple of days ago. 2) I don''t have a support contract. 3) I''m really busy trying to solve the issue with my board.> > As said above - it''s supposed to work (both with the 3.3.1 that SLE11 > ships with [or really, with the most recent maintenance updates in > place for both kernel and Xen] and upstream versions down to 3.2.0). > The only limitation I know about for certain (Sun?) systems is that > not all interrupts may be usable by their devices/drivers. That > limitation goes away only with 4.0 (i.e. earlier upstream versions > won''t - afaict - get you anywhere either if this is the case).So we should be able use 4.0 on SLES 11 somehow? How can I install a newer Xen on this system. The only way that I can get anything to work that I compile from source is using the Xen 3.3.1 SRPM and using rpmbuild to build the RPMs from that. With that I can also modify the 3.3.1 Xen a bit to add printfs and the like to actually debug the problem with my board (although that process is a total pain). Is it possible to build something from the hg Xen repo and install it on the SLES 11 system and have it work? There seem to be a lot of suse specific patches in the SRPMs, but I haven''t gone through them to see what they are all about... thanks dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> Dan Gora <dan.gora@gmail.com> 16.03.10 12:48 >>> >On Tue, Mar 16, 2010 at 7:17 AM, Jan Beulich <JBeulich@novell.com> wrote: >> If so, why didn''t you report this to our support folks yet? > >I haven''t reported the 3.4.1 crash just because: >1) I just found it a couple of days ago. >2) I don''t have a support contract. >3) I''m really busy trying to solve the issue with my board.Reporting 3.4.1 problems there wouldn''t be accepted anyway. This was just under the (now known wrong) assumption that the shipped hypervisor also wouldn''t work.>So we should be able use 4.0 on SLES 11 somehow? How can I install a >newer Xen on this system. The only way that I can get anything to >work that I compile from source is using the Xen 3.3.1 SRPM and using >rpmbuild to build the RPMs from that. With that I can also modify the >3.3.1 Xen a bit to add printfs and the like to actually debug the >problem with my board (although that process is a total pain).I think you better forget rpm here. Extract the Xen tarball and run "make xen tools" followed by (as root) "make install-xen" and possibly (if you don''t care overwriting files coming from the Xen RPMs, and if you''re prepared to sort out eventual inconsistencies) "make install-tools" (I generally avoid doing the latter).>Is it possible to build something from the hg Xen repo and install it >on the SLES 11 system and have it work? There seem to be a lot of >suse specific patches in the SRPMs, but I haven''t gone through them to >see what they are all about...Quite a number of them are backports of upstream changesets. But of course there are also (too) many that don''t have upstream equivalents. Nevertheless, running newer upstream Xen underneath SLE11 (or other SuSE) kernels is supposed to work (I occasionally find myself needing to do this). Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel