Hi: I am trying to run xen in a 32-way system but it hangs when booting. If I reduce the number of processors to 16, everything works fine. The white paper in xensource.com says that it supports up to 32 real procesors, so i am a little confused abou it. Any idea? Can it be the amount of RAM memory? is there a limit? the machine is x86_64 and it has 64GB of ram memory. Thanks Pablo _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, May 30, 2006 at 11:36:09AM -0700, Pablo Montesinos wrote:> I am trying to run xen in a 32-way system but it hangs when booting. If I > reduce the number of processors to 16, everything works fine. The white > paper in [1]xensource.com says that it supports up to 32 real procesors, > so i am a little confused abou it. Any idea? Can it be the amount of RAM > memory? is there a limit? the machine is x86_64 and it has 64GB of ram > memory.Does Xen print anything on the console before hanging? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
yes (XEN) CPU#31 had -112556 usecs TSC skew, fixed it up (XEN) Cyclone: Could not find valid CBAR value (XEN) Platform timer is 333.333 Mhz HPET (XEN) Brought up 32 CPUs and then it stays there forever On 5/30/06, Emmanuel Ackaouy <ack@xensource.com> wrote:> > On Tue, May 30, 2006 at 11:36:09AM -0700, Pablo Montesinos wrote: > > I am trying to run xen in a 32-way system but it hangs when booting. > If I > > reduce the number of processors to 16, everything works fine. The > white > > paper in [1]xensource.com says that it supports up to 32 real > procesors, > > so i am a little confused abou it. Any idea? Can it be the amount of > RAM > > memory? is there a limit? the machine is x86_64 and it has 64GB of > ram > > memory. > > Does Xen print anything on the console before hanging? >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Actually, there is some skew in all the processors. Pablo On 5/30/06, Uros Trebec <uros.trebec@gmail.com> wrote:> > On 5/30/06, Pablo Montesinos <pablomontesinos@gmail.com> wrote: > > yes > > > > (XEN) CPU#31 had -112556 usecs TSC skew, fixed it up > > It looks like there''s something wrong with processor #31... > Try removing it and see what happens. > > regards, > Uros >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
What kind of 32-way system is this? I have been able to run Xen 3.0.2 on a 64-way Unisys ES7000 without any issues. I am cross-posting on the xen-devel list in case it is an IBM machine. Could you please send more serial console output? Thanks, Aravindh ________________________________ From: Pablo Montesinos [mailto:pablomontesinos@gmail.com] Sent: Tuesday, May 30, 2006 5:06 PM To: Uros Trebec; xen-users@lists.xensource.com Subject: Re: [Xen-users] Xen in a 32 way system Actually, there is some skew in all the processors. Pablo On 5/30/06, Uros Trebec <uros.trebec@gmail.com > wrote: On 5/30/06, Pablo Montesinos < pablomontesinos@gmail.com <mailto:pablomontesinos@gmail.com> > wrote:> yes > > (XEN) CPU#31 had -112556 usecs TSC skew, fixed it upIt looks like there''s something wrong with processor #31... Try removing it and see what happens. regards, Uros _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
And what is the underlying host OS??? In other words, what os is Domain0 running when you are running Xen??? there could be a limitation caused by the underlying OS that is causing this... and as asked before, what kind of machine are you running this on?? Jeff _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
* Puthiyaparambil, Aravindh <aravindh.puthiyaparambil@unisys.com> [2006-05-31 09:00]:> What kind of 32-way system is this? I have been able to run Xen 3.0.2 on > a 64-way Unisys ES7000 without any issues. I am cross-posting on the > xen-devel list in case it is an IBM machine. Could you please send more > serial console output? > > > > Thanks, > > Aravindh > > > > ________________________________ > > From: Pablo Montesinos [mailto:pablomontesinos@gmail.com] > Sent: Tuesday, May 30, 2006 5:06 PM > To: Uros Trebec; xen-users@lists.xensource.com > Subject: Re: [Xen-users] Xen in a 32 way system > > > > Actually, there is some skew in all the processors. > Pablo > > On 5/30/06, Uros Trebec <uros.trebec@gmail.com > wrote: > > On 5/30/06, Pablo Montesinos < pablomontesinos@gmail.com > <mailto:pablomontesinos@gmail.com> > wrote: > > yes > > > > (XEN) CPU#31 had -112556 usecs TSC skew, fixed it upAFAIK, this is harmless. Our x460 32-way spits out the same information. This is part of the TSC synchronization code in Xen during boot. (XEN) checking TSC synchronization across 32 CPUs: (XEN) CPU#0 had -125003 usecs TSC skew, fixed it up. (XEN) CPU#1 had -125001 usecs TSC skew, fixed it up. (XEN) CPU#2 had -124997 usecs TSC skew, fixed it up. (XEN) CPU#3 had -125001 usecs TSC skew, fixed it up. (XEN) CPU#4 had -125000 usecs TSC skew, fixed it up. (XEN) CPU#5 had -125000 usecs TSC skew, fixed it up. (XEN) CPU#6 had -124992 usecs TSC skew, fixed it up. (XEN) CPU#7 had -124976 usecs TSC skew, fixed it up. (XEN) CPU#8 had -125022 usecs TSC skew, fixed it up. (XEN) CPU#9 had -125022 usecs TSC skew, fixed it up. (XEN) CPU#10 had -125022 usecs TSC skew, fixed it up. (XEN) CPU#11 had -125022 usecs TSC skew, fixed it up. (XEN) CPU#12 had -125020 usecs TSC skew, fixed it up. (XEN) CPU#13 had -125020 usecs TSC skew, fixed it up. (XEN) CPU#14 had -125019 usecs TSC skew, fixed it up. (XEN) CPU#15 had -125019 usecs TSC skew, fixed it up. (XEN) CPU#16 had -124498 usecs TSC skew, fixed it up. (XEN) CPU#17 had -124498 usecs TSC skew, fixed it up. (XEN) CPU#18 had -124498 usecs TSC skew, fixed it up. (XEN) CPU#19 had -124498 usecs TSC skew, fixed it up. (XEN) CPU#20 had -124498 usecs TSC skew, fixed it up. (XEN) CPU#21 had -124498 usecs TSC skew, fixed it up. (XEN) CPU#22 had -124496 usecs TSC skew, fixed it up. (XEN) CPU#23 had -124496 usecs TSC skew, fixed it up. (XEN) CPU#24 had 374521 usecs TSC skew, fixed it up. (XEN) CPU#25 had 374520 usecs TSC skew, fixed it up. (XEN) CPU#26 had 374511 usecs TSC skew, fixed it up. (XEN) CPU#27 had 374511 usecs TSC skew, fixed it up. (XEN) CPU#28 had 374511 usecs TSC skew, fixed it up. (XEN) CPU#29 had 374511 usecs TSC skew, fixed it up. (XEN) CPU#30 had 374521 usecs TSC skew, fixed it up. (XEN) CPU#31 had 374520 usecs TSC skew, fixed it up. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yeah, it''s an IBM machine, 32 way, xeon x86_64, 64GB of ram. DomO: redhat enterprise linux DomU: Fedora core 5 Thanks, Pablo On 5/31/06, Puthiyaparambil, Aravindh <aravindh.puthiyaparambil@unisys.com> wrote:> > What kind of 32-way system is this? I have been able to run Xen 3.0.2 on > a 64-way Unisys ES7000 without any issues. I am cross-posting on the > xen-devel list in case it is an IBM machine. Could you please send more > serial console output? > > > > Thanks, > > Aravindh > > > ------------------------------ > > *From:* Pablo Montesinos [mailto:pablomontesinos@gmail.com] > *Sent:* Tuesday, May 30, 2006 5:06 PM > *To:* Uros Trebec; xen-users@lists.xensource.com > *Subject:* Re: [Xen-users] Xen in a 32 way system > > > > Actually, there is some skew in all the processors. > Pablo > > On 5/30/06, *Uros Trebec* <uros.trebec@gmail.com > wrote: > > On 5/30/06, Pablo Montesinos < pablomontesinos@gmail.com> wrote: > > yes > > > > (XEN) CPU#31 had -112556 usecs TSC skew, fixed it up > > It looks like there''s something wrong with processor #31... > Try removing it and see what happens. > > regards, > Uros > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
You said you were able to boot the same box as a 16-way SMP? How did you do that exactly? Did you pass maxcpus=16 on the command line or did you disable the CPUs somehow? And when you say the box hangs... Can you eventually ping it? On Wed, May 31, 2006 at 09:07:15AM -0700, Pablo Montesinos wrote:> Yeah, it''s an IBM machine, 32 way, xeon x86_64, 64GB of ram. > DomO: redhat enterprise linux > DomU: Fedora core 5 > > Thanks, > Pablo > > On 5/31/06, Puthiyaparambil, Aravindh > <[1]aravindh.puthiyaparambil@unisys.com> wrote: > > What kind of 32-way system is this? I have been able to run Xen 3.0.2 on > a 64-way Unisys ES7000 without any issues. I am cross-posting on the > xen-devel list in case it is an IBM machine. Could you please send more > serial console output? > > > > Thanks, > > Aravindh > > > > -------------------------------------------------------------------------- > > From: Pablo Montesinos [mailto:[2]pablomontesinos@gmail.com] > Sent: Tuesday, May 30, 2006 5:06 PM > To: Uros Trebec; [3]xen-users@lists.xensource.com > Subject: Re: [Xen-users] Xen in a 32 way system > > > > Actually, there is some skew in all the processors. > Pablo > > On 5/30/06, Uros Trebec <[4]uros.trebec@gmail.com > wrote: > > On 5/30/06, Pablo Montesinos < [5]pablomontesinos@gmail.com > wrote: > > yes > > > > (XEN) CPU#31 had -112556 usecs TSC skew, fixed it up > > It looks like there''s something wrong with processor #31... > Try removing it and see what happens. > > regards, > Uros > > > > References > > Visible links > 1. mailto:aravindh.puthiyaparambil@unisys.com > 2. mailto:pablomontesinos@gmail.com > 3. mailto:xen-users@lists.xensource.com > 4. mailto:uros.trebec@gmail.com > 5. mailto:pablomontesinos@gmail.com> _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
This machine can be configured as a 4-8-16-32 way system. When I disable 16 CPUs, it freezes while loading xen, before it starts showing the boot output for the Domain0 linux. Thanks, Pablo On 5/31/06, Emmanuel Ackaouy <ack@xensource.com> wrote:> > You said you were able to boot the same box as a 16-way SMP? > How did you do that exactly? Did you pass maxcpus=16 on the > command line or did you disable the CPUs somehow? > > And when you say the box hangs... Can you eventually ping > it? > > On Wed, May 31, 2006 at 09:07:15AM -0700, Pablo Montesinos wrote: > > Yeah, it''s an IBM machine, 32 way, xeon x86_64, 64GB of ram. > > DomO: redhat enterprise linux > > DomU: Fedora core 5 > > > > Thanks, > > Pablo > > > > On 5/31/06, Puthiyaparambil, Aravindh > > <[1]aravindh.puthiyaparambil@unisys.com> wrote: > > > > What kind of 32-way system is this? I have been able to run Xen > 3.0.2 on > > a 64-way Unisys ES7000 without any issues. I am cross-posting on > the > > xen-devel list in case it is an IBM machine. Could you please send > more > > serial console output? > > > > > > > > Thanks, > > > > Aravindh > > > > > > > > > -------------------------------------------------------------------------- > > > > From: Pablo Montesinos [mailto:[2]pablomontesinos@gmail.com] > > Sent: Tuesday, May 30, 2006 5:06 PM > > To: Uros Trebec; [3]xen-users@lists.xensource.com > > Subject: Re: [Xen-users] Xen in a 32 way system > > > > > > > > Actually, there is some skew in all the processors. > > Pablo > > > > On 5/30/06, Uros Trebec <[4]uros.trebec@gmail.com > wrote: > > > > On 5/30/06, Pablo Montesinos < [5]pablomontesinos@gmail.com > > wrote: > > > yes > > > > > > (XEN) CPU#31 had -112556 usecs TSC skew, fixed it up > > > > It looks like there''s something wrong with processor #31... > > Try removing it and see what happens. > > > > regards, > > Uros > > > > > > > > References > > > > Visible links > > 1. mailto:aravindh.puthiyaparambil@unisys.com > > 2. mailto:pablomontesinos@gmail.com > > 3. mailto:xen-users@lists.xensource.com > > 4. mailto:uros.trebec@gmail.com > > 5. mailto:pablomontesinos@gmail.com > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Sorry, I meant when I enable the last 16 cpus. Sorry for the spam On 5/31/06, Pablo Montesinos <pablomontesinos@gmail.com> wrote:> > This machine can be configured as a 4-8-16-32 way system. > When I disable 16 CPUs, it freezes while loading xen, before > it starts showing the boot output for the Domain0 linux. > Thanks, Pablo > > > > On 5/31/06, Emmanuel Ackaouy <ack@xensource.com> wrote: > > > > You said you were able to boot the same box as a 16-way SMP? > > How did you do that exactly? Did you pass maxcpus=16 on the > > command line or did you disable the CPUs somehow? > > > > And when you say the box hangs... Can you eventually ping > > it? > > > > On Wed, May 31, 2006 at 09:07:15AM -0700, Pablo Montesinos wrote: > > > Yeah, it''s an IBM machine, 32 way, xeon x86_64, 64GB of ram. > > > DomO: redhat enterprise linux > > > DomU: Fedora core 5 > > > > > > Thanks, > > > Pablo > > > > > > On 5/31/06, Puthiyaparambil, Aravindh > > > <[1]aravindh.puthiyaparambil@unisys.com> wrote: > > > > > > What kind of 32-way system is this? I have been able to run Xen > > 3.0.2 on > > > a 64-way Unisys ES7000 without any issues. I am cross-posting on > > the > > > xen-devel list in case it is an IBM machine. Could you please > > send more > > > serial console output? > > > > > > > > > > > > Thanks, > > > > > > Aravindh > > > > > > > > > > > > > > -------------------------------------------------------------------------- > > > > > > From: Pablo Montesinos [mailto:[2]pablomontesinos@ gmail.com] > > > Sent: Tuesday, May 30, 2006 5:06 PM > > > To: Uros Trebec; [3]xen-users@lists.xensource.com > > > Subject: Re: [Xen-users] Xen in a 32 way system > > > > > > > > > > > > Actually, there is some skew in all the processors. > > > Pablo > > > > > > On 5/30/06, Uros Trebec <[4]uros.trebec@gmail.com > wrote: > > > > > > On 5/30/06, Pablo Montesinos < [5]pablomontesinos@gmail.com > > > wrote: > > > > yes > > > > > > > > (XEN) CPU#31 had -112556 usecs TSC skew, fixed it up > > > > > > It looks like there''s something wrong with processor #31... > > > Try removing it and see what happens. > > > > > > regards, > > > Uros > > > > > > > > > > > > References > > > > > > Visible links > > > 1. mailto:aravindh.puthiyaparambil@unisys.com > > > 2. mailto:pablomontesinos@gmail.com > > > 3. mailto:xen-users@lists.xensource.com > > > 4. mailto:uros.trebec@gmail.com > > > 5. mailto: pablomontesinos@gmail.com > > > > > _______________________________________________ > > > Xen-users mailing list > > > Xen-users@lists.xensource.com > > > http://lists.xensource.com/xen-users > > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Can you please send the serial output? I don''t think anyone can help without that. Regards, Aravindh ________________________________ From: Pablo Montesinos [mailto:pablomontesinos@gmail.com] Sent: Wednesday, May 31, 2006 12:17 PM To: Pablo Montesinos; Puthiyaparambil, Aravindh; xen-users@lists.xensource.com; ryanh@us.ibm.com Subject: Re: [Xen-users] Xen in a 32 way system This machine can be configured as a 4-8-16-32 way system. When I disable 16 CPUs, it freezes while loading xen, before it starts showing the boot output for the Domain0 linux. Thanks, Pablo On 5/31/06, Emmanuel Ackaouy <ack@xensource.com> wrote: You said you were able to boot the same box as a 16-way SMP? How did you do that exactly? Did you pass maxcpus=16 on the command line or did you disable the CPUs somehow? And when you say the box hangs... Can you eventually ping it? On Wed, May 31, 2006 at 09:07:15AM -0700, Pablo Montesinos wrote:> Yeah, it''s an IBM machine, 32 way, xeon x86_64, 64GB of ram. > DomO: redhat enterprise linux > DomU: Fedora core 5 > > Thanks, > Pablo > > On 5/31/06, Puthiyaparambil, Aravindh > <[1]aravindh.puthiyaparambil@unisys.com> wrote: > > What kind of 32-way system is this? I have been able to run Xen3.0.2 on> a 64-way Unisys ES7000 without any issues. I am cross-posting onthe> xen-devel list in case it is an IBM machine. Could you pleasesend more> serial console output? > > > > Thanks, > > Aravindh > > > >------------------------------------------------------------------------ --> > From: Pablo Montesinos [mailto:[2]pablomontesinos@ gmail.com] > Sent: Tuesday, May 30, 2006 5:06 PM > To: Uros Trebec; [3]xen-users@lists.xensource.com > Subject: Re: [Xen-users] Xen in a 32 way system > > > > Actually, there is some skew in all the processors. > Pablo > > On 5/30/06, Uros Trebec <[4]uros.trebec@gmail.com > wrote: > > On 5/30/06, Pablo Montesinos < [5]pablomontesinos@gmail.com >wrote:> > yes > > > > (XEN) CPU#31 had -112556 usecs TSC skew, fixed it up > > It looks like there''s something wrong with processor #31... > Try removing it and see what happens. > > regards, > Uros > > > > References > > Visible links > 1. mailto:aravindh.puthiyaparambil@unisys.com > 2. mailto:pablomontesinos@gmail.com > 3. mailto:xen-users@lists.xensource.com > 4. mailto:uros.trebec@gmail.com > 5. mailto: pablomontesinos@gmail.com<mailto:pablomontesinos@gmail.com>> _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
* Pablo Montesinos <pablomontesinos@gmail.com> [2006-05-31 11:18]:> This machine can be configured as a 4-8-16-32 way system. > When I disable 16 CPUs, it freezes while loading xen, before > it starts showing the boot output for the Domain0 linux. > Thanks, Pablo > > > On 5/31/06, Emmanuel Ackaouy <ack@xensource.com> wrote: > > > >You said you were able to boot the same box as a 16-way SMP? > >How did you do that exactly? Did you pass maxcpus=16 on the > >command line or did you disable the CPUs somehow? > > > >And when you say the box hangs... Can you eventually ping > >it? > >I''ve booted our 4-node 32-way as a 2-node fine. I think we are going to need to see a serial dump of the failure to get any idea of what is going on. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 5/31/06, Pablo Montesinos <pablomontesinos@gmail.com> wrote:> Yeah, it''s an IBM machine, 32 way, xeon x86_64, 64GB of ram. > DomO: redhat enterprise linux > DomU: Fedora core 5 > > Thanks, > PabloRegarding the Skew, I agree that it is harmless... I have seen that on several x460 permutations as well, and it does not seem to affect anything. However, have you tried passing something like apic=bigsmp (works on SLES on a similar configuration)??? I know this is FC and not SLES or SuSE, or RHEL for that matter, but to me, this sounds a lot like the Xen kernel being used with FC5 is not enabling bigsmp by default... I saw the same thing elsewhere and passing apic=bigsmp seems to fix the problem for me... so your milage may vary... Jeff _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
For what its worth, I have not tried Xen with FC5.. no time to do so... BUT I have run it with SLES successfully... If I understand the process correctly, to use Xen you need a xen aware kernel... in SLES there is a standard SLES kernel and a Xen aware SLES kernel...and you must boot the SLES Xen kernel to run Xend and have Xen functionality... Now, supposing that it is different for FC5 and the FC5 kernel itself IS xen aware, instead of going the SLES route and having seperate kernels (which would be a surprise as Red Hat seems to just love spinning a new kernel for every different feature), it still seems to me that the kernel you are booting is not set up to handle > 16 processors... that was fixed by Red Hat and the Linux community with the bigsmp stuff... that is why you can boot when you limit cpus to 16, but not when you try booting 32... if apic=bigsmp does not work on FC5, then there may be an FC5 equiv boot parameter that will enable bigsmp, IF the support is there in the FC5 kernel... Like I said previously, I have only tried this with SLES, so YMMV.... Jeff On 5/31/06, Pablo Montesinos <pablomontesinos@gmail.com> wrote:> But the problems I am experiencing are before I see dom0 boot, so I doubt > it''s has anything to do with fedora core, no? > > Thanks, > pablo_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pablo Montesinos
2006-May-31 18:40 UTC
Re: [Xen-devel] RE: [Xen-users] Xen in a 32 way system
Hi all: I have attached the serial output for the system with 32 procs. To recap: It does boot with 4,8,16 procs. I have domains kiking and running if I boot with only 4 processors. if I use 8 or 16 procs, xend kills the network when I launch it, so I am also looking into it now. Thanks, Pablo _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Puthiyaparambil, Aravindh
2006-May-31 19:38 UTC
RE: [Xen-devel] RE: [Xen-users] Xen in a 32 way system
>From a quick code view it looks like before "domain_create(0, 0)"[setup.c] could complete an serial console interrupt occurred. But I don''t understand how construct_dom0() could be called before domain_create() is completed. Hopefully Ryan can shine more light on this. Aravindh> -----Original Message----- > From: Pablo Montesinos [mailto:pablomontesinos@gmail.com] > Sent: Wednesday, May 31, 2006 2:40 PM > To: Ryan Harper > Cc: Puthiyaparambil, Aravindh; Uros Trebec;xen-users@lists.xensource.com;> xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] RE: [Xen-users] Xen in a 32 way system > > Hi all: > I have attached the serial output for the system with 32 procs. > To recap: It does boot with 4,8,16 procs. > > > I have domains kiking and running if I boot with only 4 processors. > if I use 8 or 16 procs, xend kills the network when I launch it, > so I am also looking into it now. > > Thanks, > Pablo_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
* Pablo Montesinos <pablomontesinos@gmail.com> [2006-05-31 13:41]:> (XEN) Total of 32 processors activated. > (XEN) ENABLING IO-APIC IRQs > (XEN) ..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1 > (XEN) ..MP-BIOS bug: 8254 timer not connected to IO-APIC > (XEN) ...trying to set up timer (IRQ0) through the 8259A ... failed. > (XEN) ...trying to set up timer as Virtual Wire IRQ... works.I think you need to use the "new ACK" method. When my x460 boots, the above section looks like this: (XEN) Total of 32 processors activated. (XEN) ENABLING IO-APIC IRQs (XEN) -> Using new ACK method (XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1 (XEN) ..MP-BIOS bug: 8254 timer not connected to IO-APIC (XEN) ...trying to set up timer (IRQ0) through the 8259A ... failed. (XEN) ...trying to set up timer as Virtual Wire IRQ... works. I think if you add ''ioapic_ack=new'' to your xen command line in grub. I''m not sure if this is available in the testing tree or only in unstable. The other thing to try is seeing if the machine boots on xen-unstable. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pablo Montesinos
2006-May-31 22:54 UTC
Re: [Xen-devel] RE: [Xen-users] Xen in a 32 way system
I tried the "new ACK" method, no luck. I''ll try xen-unstable. Each reboot takes forever, I am going nuts :( thanks, Pablo On 5/31/06, Ryan Harper <ryanh@us.ibm.com> wrote:> * Pablo Montesinos <pablomontesinos@gmail.com> [2006-05-31 13:41]: > > > (XEN) Total of 32 processors activated. > > (XEN) ENABLING IO-APIC IRQs > > (XEN) ..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1 > > (XEN) ..MP-BIOS bug: 8254 timer not connected to IO-APIC > > (XEN) ...trying to set up timer (IRQ0) through the 8259A ... failed. > > (XEN) ...trying to set up timer as Virtual Wire IRQ... works. > > I think you need to use the "new ACK" method. When my x460 boots, the > above section looks like this: > > (XEN) Total of 32 processors activated. > (XEN) ENABLING IO-APIC IRQs > (XEN) -> Using new ACK method > (XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1 > (XEN) ..MP-BIOS bug: 8254 timer not connected to IO-APIC > (XEN) ...trying to set up timer (IRQ0) through the 8259A ... failed. > (XEN) ...trying to set up timer as Virtual Wire IRQ... works. > > I think if you add ''ioapic_ack=new'' to your xen command line in grub. > I''m not sure if this is available in the testing tree or only in > unstable. The other thing to try is seeing if the machine boots on > xen-unstable. > > > -- > Ryan Harper > Software Engineer; Linux Technology Center > IBM Corp., Austin, Tx > (512) 838-9253 T/L: 678-9253 > ryanh@us.ibm.com >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 31 May 2006, at 20:48, Ryan Harper wrote:> I think if you add ''ioapic_ack=new'' to your xen command line in grub. > I''m not sure if this is available in the testing tree or only in > unstable. The other thing to try is seeing if the machine boots on > xen-unstable.The lack of that line of output indicates that the Xen image is too old to have the new ack''ing method. 3.0 series has had teh new code for a while now. Time to upgrade to 3.0.3! -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pablo Montesinos
2006-Jun-01 16:11 UTC
Re: [Xen-devel] RE: [Xen-users] Xen in a 32 way system
I installed xen-unstable and it seems to solve that problem. However,
I am getting a kernel bug. xen-unstable downloaded the 2.6.16.13
kernel. It is possible to use a 2.6.16?
Thanks,
Pablo
PS: if you enjoy kernel bugs:
Kernel BUG at drivers/xen/core/smpboot.c:224
invalid opcode: 0000 [1] SMP
CPU 0
Modules linked in:
Pid: 1, comm: swapper Not tainted 2.6.16.13-xen #2
RIP: e030:[<ffffffff802c5d15>]
<ffffffff802c5d15>{cpu_initialize_context+514}
RSP: e02b:ffff8800018bca78 EFLAGS: 00010282
RAX: ffffffffffffffea RBX: ffff8800018bca88 RCX: 0000000000000101
RDX: ffff8800018bca88 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff8800018bdee8 R08: ffff8800102b5a20 R09: 0000000000000000
R10: ffff8800018bca88 R11: 0000000000000200 R12: 0000000000000020
R13: 0000000000000140 R14: ffff8800102b5860 R15: ffffffff8040be40
FS: 0000000000000000(0000) GS:ffffffff8052f000(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000
Process swapper (pid: 1, threadinfo ffff8800018bc000, task ffff8800018a4760)
Stack: 00000000002fab4f 0000000000000020 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000
Call Trace: <ffffffff802c5adb>{cpu_bringup+0}
<ffffffff8010b758>{divide_error+0}
<ffffffff80398fa4>{debug+0} <ffffffff80398fc0>{int3+0}
<ffffffff8010b65c>{overflow+0} <ffffffff8010b678>{bounds+0}
<ffffffff8010b694>{invalid_op+0}
<ffffffff8010b640>{device_not_available+0}
<ffffffff8010b6b0>{coprocessor_segment_overrun+0}
<ffffffff8010b6e8>{invalid_TSS+0}
<ffffffff8010b704>{segment_not_present+0}
<ffffffff8010b720>{stack_segment+0}
<ffffffff80398fdc>{general_protection+0}
<ffffffff80398f88>{page_fault+0}
<ffffffff8010b774>{spurious_interrupt_bug+0}
<ffffffff8010b608>{coprocessor_error+0}
<ffffffff8010b73c>{alignment_check+0}
<ffffffff8010b624>{simd_coprocessor_error+0}
<ffffffff8011b2e8>{ia32_syscall+0}
<ffffffff8010b2a4>{hypervisor_callback+0}
<ffffffff8010b3a4>{failsafe_callback+0}
<ffffffff8010aa94>{system_call+0}
<ffffffff80564e5f>{smp_prepare_cpus+456}
<ffffffff80108084>{init+60}
<ffffffff8010b532>{child_rip+8}
<ffffffff80275580>{acpi_ds_init_one_object+0}
<ffffffff80108048>{init+0} <ffffffff8010b52a>{child_rip+0}
Code: 0f 0b 68 e2 61 3d 80 c2 e0 00 48 81 c4 48 14 00 00 5b 41 5c
RIP <ffffffff802c5d15>{cpu_initialize_context+514} RSP
<ffff8800018bca78>
<0>Kernel panic - not syncing: Attempted to kill init!
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
(XEN) Reboot disabled on cmdline: require manual reset
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Petersson, Mats
2006-Jun-01 17:00 UTC
RE: [Xen-devel] RE: [Xen-users] Xen in a 32 way system
The line you have the BUG in is a VCPUOP_initialize, which ends up in .../xen/common/domain.c: do_vcpu_op(), where it does essentially the first thing is a check: if ((vcpuid < 0) || (vcpuid >= MAX_VIRT_CPUS)) return -EINVAL; And I think this check will fail if vcpuid is 32, unless MAX_VIRT_CPUS is not the standard value in . RSI is 32 (0x20) As per this code: ffff830000106d8c: 83 fe 1f cmp $0x1f,%esi ffff830000106d8f: 41 89 f4 mov %esi,%r12d ffff830000106d92: 49 89 d5 mov %rdx,%r13 ffff830000106d95: 48 8b 58 10 mov 0x10(%rax),%rbx ffff830000106d99: 48 c7 c0 ea ff ff ff mov $0xffffffffffffffea,%rax ffff830000106da0: 0f 87 9d 01 00 00 ja ffff830000106f43 <do_vcpu_op+0x1f3> So it seems like you''re trying to start MORE than 32 CPUS to me... Don''t know why... Or is this check bogus? It doesn''t look bogus to me... By the way, this is a "bug" in XEN-code inside the linux-kernel, so it''s not a because of the general 2.6.16.13 kernel code, and I suspect you''d hit the same thing with a slightly earlier kernel - although this code is something I haven''t looked much at before - maybe it silently ignored error-returns from this call before... -- Mats> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > Pablo Montesinos > Sent: 01 June 2006 17:11 > To: Ryan Harper > Cc: xen-devel@lists.xensource.com; Uros Trebec; > Puthiyaparambil, Aravindh; xen-users@lists.xensource.com > Subject: Re: [Xen-devel] RE: [Xen-users] Xen in a 32 way system > > I installed xen-unstable and it seems to solve that problem. > However, I am getting a kernel bug. xen-unstable downloaded > the 2.6.16.13 kernel. It is possible to use a 2.6.16? > > Thanks, > Pablo > > PS: if you enjoy kernel bugs: > > > Kernel BUG at drivers/xen/core/smpboot.c:224 invalid opcode: > 0000 [1] SMP CPU 0 Modules linked in: > Pid: 1, comm: swapper Not tainted 2.6.16.13-xen #2 > RIP: e030:[<ffffffff802c5d15>] > <ffffffff802c5d15>{cpu_initialize_context+514} > RSP: e02b:ffff8800018bca78 EFLAGS: 00010282 > RAX: ffffffffffffffea RBX: ffff8800018bca88 RCX: 0000000000000101 > RDX: ffff8800018bca88 RSI: 0000000000000020 RDI: 0000000000000000 > RBP: ffff8800018bdee8 R08: ffff8800102b5a20 R09: 0000000000000000 > R10: ffff8800018bca88 R11: 0000000000000200 R12: 0000000000000020 > R13: 0000000000000140 R14: ffff8800102b5860 R15: ffffffff8040be40 > FS: 0000000000000000(0000) GS:ffffffff8052f000(0000) > knlGS:0000000000000000 > CS: e033 DS: 0000 ES: 0000 > Process swapper (pid: 1, threadinfo ffff8800018bc000, task > ffff8800018a4760) > Stack: 00000000002fab4f 0000000000000020 0000000000000000 > 0000000000000000 > 0000000000000000 0000000000000000 0000000000000000 > 0000000000000000 > 0000000000000000 0000000000000000 Call Trace: > <ffffffff802c5adb>{cpu_bringup+0} <ffffffff8010b758>{divide_error+0} > <ffffffff80398fa4>{debug+0} <ffffffff80398fc0>{int3+0} > <ffffffff8010b65c>{overflow+0} <ffffffff8010b678>{bounds+0} > <ffffffff8010b694>{invalid_op+0} > <ffffffff8010b640>{device_not_available+0} > <ffffffff8010b6b0>{coprocessor_segment_overrun+0} > <ffffffff8010b6e8>{invalid_TSS+0} > <ffffffff8010b704>{segment_not_present+0} > <ffffffff8010b720>{stack_segment+0} > <ffffffff80398fdc>{general_protection+0} > <ffffffff80398f88>{page_fault+0} > <ffffffff8010b774>{spurious_interrupt_bug+0} > <ffffffff8010b608>{coprocessor_error+0} > <ffffffff8010b73c>{alignment_check+0} > <ffffffff8010b624>{simd_coprocessor_error+0} > <ffffffff8011b2e8>{ia32_syscall+0} > <ffffffff8010b2a4>{hypervisor_callback+0} > <ffffffff8010b3a4>{failsafe_callback+0} > <ffffffff8010aa94>{system_call+0} > <ffffffff80564e5f>{smp_prepare_cpus+456} > <ffffffff80108084>{init+60} > <ffffffff8010b532>{child_rip+8} > <ffffffff80275580>{acpi_ds_init_one_object+0} > <ffffffff80108048>{init+0} <ffffffff8010b52a>{child_rip+0} > > Code: 0f 0b 68 e2 61 3d 80 c2 e0 00 48 81 c4 48 14 00 00 5b > 41 5c RIP <ffffffff802c5d15>{cpu_initialize_context+514} RSP > <ffff8800018bca78> <0>Kernel panic - not syncing: Attempted > to kill init! > (XEN) Domain 0 crashed: rebooting machine in 5 seconds. > (XEN) Reboot disabled on cmdline: require manual reset > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 1 Jun 2006, at 18:00, Petersson, Mats wrote:> The line you have the BUG in is a VCPUOP_initialize, which ends up in > .../xen/common/domain.c: do_vcpu_op(), where it does essentially the > first thing is a check: > if ((vcpuid < 0) || (vcpuid >= MAX_VIRT_CPUS)) > return -EINVAL; > > And I think this check will fail if vcpuid is 32, unless MAX_VIRT_CPUS > is not the standard value in . RSI is 32 (0x20)Our smpboot.c in Linux should only try to initialise context for cpus in cpu_possible_map. And that map is initialised by querying Xen about the state of each VCPU. So it''s unclear how you can end up getting this crash. Looks like it needs some more tracing added... -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pablo Montesinos
2006-Jun-01 17:30 UTC
Re: [Xen-devel] RE: [Xen-users] Xen in a 32 way system
The reason why I wanted to use a 2.6.16 kernel is the following. When I installed this morning the "vanilla unstable" version I downloaded, it created only one kernel, 2.6.16.13-xen. I booted the system with it, and it didn''t work well because of some network card, but RHEL acutally booted. I didnt check the output while it booted, but I assumed that it was using all the cpus. So I reconfigured the kernel with my kernel config, and I got the crash. I am inserting some extra code in XEN to check how many cpus it is seeing. But this is the first time I look into XEN''s code, so I am going very slow. Thanks, Pablo On 6/1/06, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:> > On 1 Jun 2006, at 18:00, Petersson, Mats wrote: > > > The line you have the BUG in is a VCPUOP_initialize, which ends up in > > .../xen/common/domain.c: do_vcpu_op(), where it does essentially the > > first thing is a check: > > if ((vcpuid < 0) || (vcpuid >= MAX_VIRT_CPUS)) > > return -EINVAL; > > > > And I think this check will fail if vcpuid is 32, unless MAX_VIRT_CPUS > > is not the standard value in . RSI is 32 (0x20) > > Our smpboot.c in Linux should only try to initialise context for cpus > in cpu_possible_map. And that map is initialised by querying Xen about > the state of each VCPU. So it''s unclear how you can end up getting this > crash. Looks like it needs some more tracing added... > > -- Keir > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Petersson, Mats
2006-Jun-01 17:59 UTC
RE: [Xen-devel] RE: [Xen-users] Xen in a 32 way system
Isn''t it a bug that we don''t check for -EINVAL in the line
here:
for (i = 0; i < NR_CPUS; i++) {
rc = HYPERVISOR_vcpu_op(VCPUOP_is_up, i, NULL);
if (rc == -ENOENT)
break;
cpu_set(i, cpu_possible_map);
}
If NR_CPUS in Linux is bigger than 32, then rc = -EINVAL, not -ENOENT
when it returns from the vcpu_op, because the first check is to see if
vcpuid is >=32...
Checking if (rc < 0) would be better, I should think...
--
Mats
> -----Original Message-----
> From: Pablo Montesinos [mailto:pablomontesinos@gmail.com]
> Sent: 01 June 2006 18:30
> To: Keir Fraser
> Cc: Petersson, Mats; Uros Trebec;
> xen-devel@lists.xensource.com; Puthiyaparambil, Aravindh;
> Ryan Harper; xen-users@lists.xensource.com
> Subject: Re: [Xen-devel] RE: [Xen-users] Xen in a 32 way system
>
> The reason why I wanted to use a 2.6.16 kernel is the
> following. When I installed this morning the "vanilla
> unstable" version I downloaded, it created only one kernel,
> 2.6.16.13-xen. I booted the system with it, and it didn''t
> work well because of some network card, but RHEL acutally
> booted. I didnt check the output while it booted, but I
> assumed that it was using all the cpus. So I reconfigured the
> kernel with my kernel config, and I got the crash.
>
> I am inserting some extra code in XEN to check how many cpus
> it is seeing. But this is the first time I look into XEN''s
> code, so I am going very slow.
>
> Thanks,
> Pablo
>
> On 6/1/06, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> >
> > On 1 Jun 2006, at 18:00, Petersson, Mats wrote:
> >
> > > The line you have the BUG in is a VCPUOP_initialize,
> which ends up
> > > in
> > > .../xen/common/domain.c: do_vcpu_op(), where it does
> essentially the
> > > first thing is a check:
> > > if ((vcpuid < 0) || (vcpuid >= MAX_VIRT_CPUS))
> > > return -EINVAL;
> > >
> > > And I think this check will fail if vcpuid is 32, unless
> > > MAX_VIRT_CPUS is not the standard value in . RSI is 32 (0x20)
> >
> > Our smpboot.c in Linux should only try to initialise
> context for cpus
> > in cpu_possible_map. And that map is initialised by
> querying Xen about
> > the state of each VCPU. So it''s unclear how you can end up
getting
> > this crash. Looks like it needs some more tracing added...
> >
> > -- Keir
> >
> >
>
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
On 1 Jun 2006, at 18:59, Petersson, Mats wrote:> Isn''t it a bug that we don''t check for -EINVAL in the line here: > for (i = 0; i < NR_CPUS; i++) { > rc = HYPERVISOR_vcpu_op(VCPUOP_is_up, i, NULL); > if (rc == -ENOENT) > break; > cpu_set(i, cpu_possible_map); > } > > If NR_CPUS in Linux is bigger than 32, then rc = -EINVAL, not -ENOENT > when it returns from the vcpu_op, because the first check is to see if > vcpuid is >=32... > > Checking if (rc < 0) would be better, I should think...Yes, the current code is rather mad. I''ll fix it. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Petersson, Mats
2006-Jun-01 18:22 UTC
RE: [Xen-devel] RE: [Xen-users] Xen in a 32 way system
> -----Original Message----- > From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] > Sent: 01 June 2006 19:09 > To: Petersson, Mats > Cc: xen-devel@lists.xensource.com; Puthiyaparambil, Aravindh; > Uros Trebec; Ryan Harper; xen-users@lists.xensource.com; > Pablo Montesinos > Subject: Re: [Xen-devel] RE: [Xen-users] Xen in a 32 way system > > > On 1 Jun 2006, at 18:59, Petersson, Mats wrote: > > > Isn''t it a bug that we don''t check for -EINVAL in the line here: > > for (i = 0; i < NR_CPUS; i++) { > > rc = HYPERVISOR_vcpu_op(VCPUOP_is_up, i, NULL); > > if (rc == -ENOENT) > > break; > > cpu_set(i, cpu_possible_map); > > } > > > > If NR_CPUS in Linux is bigger than 32, then rc = -EINVAL, > not -ENOENT > > when it returns from the vcpu_op, because the first check > is to see if > > vcpuid is >=32... > > > > Checking if (rc < 0) would be better, I should think... > > Yes, the current code is rather mad. I''ll fix it.Of course, if NR_CPUS is 32, it shouldn''t matter... Pablo, can you check the CONFIG_NR_CPUS in your .config for your linux-2.6.16.13-xen? As in my system, NR_CPUS = 32.. -- Mats> > -- Keir > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pablo Montesinos
2006-Jun-01 20:38 UTC
Re: [Xen-devel] RE: [Xen-users] Xen in a 32 way system
hat tip for you... It is booting, I am still not able to run xend with more than 4 procs, but that''s a complete different story. Ryan: did you do anything special to load configure xend? it just hangs as soon as I enable more than 1 chassis Thank you very much to all of you guys! Pablo On 6/1/06, Petersson, Mats <Mats.Petersson@amd.com> wrote:> > -----Original Message----- > > From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] > > Sent: 01 June 2006 19:09 > > To: Petersson, Mats > > Cc: xen-devel@lists.xensource.com; Puthiyaparambil, Aravindh; > > Uros Trebec; Ryan Harper; xen-users@lists.xensource.com; > > Pablo Montesinos > > Subject: Re: [Xen-devel] RE: [Xen-users] Xen in a 32 way system > > > > > > On 1 Jun 2006, at 18:59, Petersson, Mats wrote: > > > > > Isn''t it a bug that we don''t check for -EINVAL in the line here: > > > for (i = 0; i < NR_CPUS; i++) { > > > rc = HYPERVISOR_vcpu_op(VCPUOP_is_up, i, NULL); > > > if (rc == -ENOENT) > > > break; > > > cpu_set(i, cpu_possible_map); > > > } > > > > > > If NR_CPUS in Linux is bigger than 32, then rc = -EINVAL, > > not -ENOENT > > > when it returns from the vcpu_op, because the first check > > is to see if > > > vcpuid is >=32... > > > > > > Checking if (rc < 0) would be better, I should think... > > > > Yes, the current code is rather mad. I''ll fix it. > > Of course, if NR_CPUS is 32, it shouldn''t matter... > > Pablo, can you check the CONFIG_NR_CPUS in your .config for your > linux-2.6.16.13-xen? As in my system, NR_CPUS = 32.. > > -- > Mats > > > > -- Keir > > > > > > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
* Pablo Montesinos <pablomontesinos@gmail.com> [2006-06-01 15:40]:> hat tip for you... It is booting, I am still not able to run xend with > more than 4 procs, but that''s a complete different story. > Ryan: did you do anything special to load configure xend? it just > hangs as soon as I enable more than 1 chassisNo, I''ve done nothing special. One of the quirks of our network is that when xend comes up, it messes with the ethernet interface names and sets up the bridging. If I manually start xend via ssh, that ssh session hangs. I don''t think that is a hardware or software issue, more of an issue with our lab infrastructure. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Puthiyaparambil, Aravindh
2006-Jun-01 21:16 UTC
RE: [Xen-devel] RE: [Xen-users] Xen in a 32 way system
> hat tip for you... It is booting, I am still not able to run xend with > more than 4 procs, but that''s a complete different story. > Ryan: did you do anything special to load configure xend? it just > hangs as soon as I enable more than 1 chassisAre you on the latest changeset? Changeset 10240 fixed a networking issue. We were having problems on our systems that did not have a default gateway defined. Though your problem could be something totally different. Aravindh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel