Keir,
This patch reverts part of the 7402 patch. The reverted part is
conversion from unsigned long to uint32_t for evtchn_pending,
evtchn_mask.
With this patch on top of the latest tip (changeset:
7502:e70ea9465b310e1cba9678ac4e9ad534bb8b670a) it can boot SMP dom0 on
x86_64.
Thanks & Regards,
Nitin
------------------------------------------------------------------------
-----------
Open Source Technology Center, Intel Corp
>-----Original Message-----
>From: Ryan Harper [mailto:ryanh@us.ibm.com]
>Sent: Thursday, October 27, 2005 4:50 PM
>To: Kamble, Nitin A
>Cc: Keir Fraser; Ian Pratt; xen-devel
>Subject: Re: [Xen-devel] RE: [PATCH] [VT]long event-channel pending and
>mask arrays
>
>* Kamble, Nitin A <nitin.a.kamble@intel.com> [2005-10-27 18:34]:
>> Keir,
>> You are right. It can be changed to long but it is not necessary.
>>
>> The issue in SMP dom0 for x86_64 got introduced before your SMP code
>> restructuring (merge smpboot.c) rev 7415.
>>
>> The rev 7412 has my fix for SMP dom0/domU. I have tested that fix
here>> which was working fine. That patch is pulled in the tree with many
other>> patches on the same day. With your 7402 patch (long masks) it is
>> breaking SMP dom0 differently. If I checkout the 7412 and then revert
>
>I checked out 7401 and added in 7412 and can confirm that x86_64 DOM0
>SMP boots.
>
>--
>Ryan Harper
>Software Engineer; Linux Technology Center
>IBM Corp., Austin, Tx
>(512) 838-9253 T/L: 678-9253
>ryanh@us.ibm.com
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
On 28 Oct 2005, at 02:05, Kamble, Nitin A wrote:> This patch reverts part of the 7402 patch. The reverted part is > conversion from unsigned long to uint32_t for evtchn_pending, > evtchn_mask.That was the original *point* of the patch, so yours is basically the anti-patch. I can take a look... what is the bad SMP dom0 behaviour you are seeing (is there a bugzilla reference)? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Puthiyaparambil, Aravindh
2005-Oct-28 14:34 UTC
RE: [Xen-devel] Re: [PATCH] SMP dom0 boot fix
Keir, I have not tried a SMP Dom0 but I have been unable to bring up any x86_64 SMP DomUs for a while now. I don''t know the symptoms of the SMP Dom0 issue so I can''t say for sure that they are connected. Here is the bug report. http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=347 FYI, x86_32 PAE SMP DomUs are working. Aravindh> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- > bounces@lists.xensource.com] On Behalf Of Keir Fraser > Sent: Friday, October 28, 2005 3:32 AM > To: Kamble, Nitin A > Cc: Ian Pratt; xen-devel; Ryan Harper > Subject: [Xen-devel] Re: [PATCH] SMP dom0 boot fix > > > On 28 Oct 2005, at 02:05, Kamble, Nitin A wrote: > > > This patch reverts part of the 7402 patch. The reverted part is > > conversion from unsigned long to uint32_t for evtchn_pending, > > evtchn_mask. > > That was the original *point* of the patch, so yours is basically the > anti-patch. > > I can take a look... what is the bad SMP dom0 behaviour you are seeing > (is there a bugzilla reference)? > > -- Keir > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-10-28 09:20]:> > On 28 Oct 2005, at 02:05, Kamble, Nitin A wrote: > > > This patch reverts part of the 7402 patch. The reverted part is > >conversion from unsigned long to uint32_t for evtchn_pending, > >evtchn_mask. > > That was the original *point* of the patch, so yours is basically the > anti-patch. > > I can take a look... what is the bad SMP dom0 behaviour you are seeing > (is there a bugzilla reference)?I''ll file a bug if Nitin doesn''t beat me to it. I was seeing smp_call_function() blocking while it waited for a function to return from being invoked on the second processor in early boot. It would block forever installing the deadline io scheduler, digging deeper, it was after the second processor was up, and the io schedulers call kmem_cache_create() in mm/slab.c, which would eventually result in a call to smp_call_function() and passed in wait=1 which forces the function to block until it has run on all online processors. WARK: CPU0 in enable_cpucache() WARK: CPU0 do_tune_cpucache() in WARK: CPU0 calling all cpus with do_ccupdate_local() WARK: CPU0 smp_call_function_all_cpus() WARK: CPU0 running do_ccupdate_local() WARK: CPU0 invoking smp_call_function() At this point send_IPI_allbutself() has been invoked and the system just sits and waits on CPU1 to run the function. But, CPU1''s evtchn_upcall_mask was set (1), so I''m guessing the pending interrupt is never acknowledged. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 28 Oct 2005, at 15:59, Ryan Harper wrote:> At this point send_IPI_allbutself() has been invoked and the system > just sits and waits on CPU1 to run the function. But, CPU1''s > evtchn_upcall_mask was set (1), so I''m guessing the pending interrupt > is never acknowledged.Okay, the good news is that''s the same bug I was able to repro last week. Turns out that CPU1''s upcall mask is getting weirdly set under its feet. Since it''s waiting on the big kernel lock, which is held by CPU0, which is waiting for acknowledgement of an interrupt in CPU1, we have a deadlock. Given the problem is in that one changeset, this can''t be hard to track down now. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 28 Oct 2005, at 16:15, Keir Fraser wrote:> > On 28 Oct 2005, at 15:59, Ryan Harper wrote: > >> At this point send_IPI_allbutself() has been invoked and the system >> just sits and waits on CPU1 to run the function. But, CPU1''s >> evtchn_upcall_mask was set (1), so I''m guessing the pending interrupt >> is never acknowledged. > > Okay, the good news is that''s the same bug I was able to repro last > week. Turns out that CPU1''s upcall mask is getting weirdly set under > its feet. Since it''s waiting on the big kernel lock, which is held by > CPU0, which is waiting for acknowledgement of an interrupt in CPU1, we > have a deadlock. > > Given the problem is in that one changeset, this can''t be hard to > track down now.Now fixed in our staging tree. sizeof_vcpu_shift in arch/xen/x86_64/xen_entry.S should be 4, not 3. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-10-28 10:41]:> > On 28 Oct 2005, at 16:15, Keir Fraser wrote: > > > > >On 28 Oct 2005, at 15:59, Ryan Harper wrote: > > > >>At this point send_IPI_allbutself() has been invoked and the system > >>just sits and waits on CPU1 to run the function. But, CPU1''s > >>evtchn_upcall_mask was set (1), so I''m guessing the pending interrupt > >>is never acknowledged. > > > >Okay, the good news is that''s the same bug I was able to repro last > >week. Turns out that CPU1''s upcall mask is getting weirdly set under > >its feet. Since it''s waiting on the big kernel lock, which is held by > >CPU0, which is waiting for acknowledgement of an interrupt in CPU1, we > >have a deadlock. > > > >Given the problem is in that one changeset, this can''t be hard to > >track down now. > > Now fixed in our staging tree. sizeof_vcpu_shift in > arch/xen/x86_64/xen_entry.S should be 4, not 3.That fixed it for me here. Thanks! -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>> Now fixed in our staging tree. sizeof_vcpu_shift in >> arch/xen/x86_64/xen_entry.S should be 4, not 3. > >That fixed it for me here. Thanks!That''s Great!!! Keir, Thanks for fixing. Thanks & Regards, Nitin ------------------------------------------------------------------------ ----------- Open Source Technology Center, Intel Corp _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks Kier. With this fix applied, I am able to boot SMP dom0. The box I am testing on is x86_64 machine with two hardware threads. However, if I turn on SMT in the Linux configuration, the kernel takes a fault in early startup (a NULL pointer reference at find_busiest_group +144). It appears that the sched domain hierarchy is not correctly set up here. Looking at the new smpboot.c, is turning on SMT support no longer valid? K. Y>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 10/28/05 11:45 am >>>On 28 Oct 2005, at 16:15, Keir Fraser wrote:> > On 28 Oct 2005, at 15:59, Ryan Harper wrote: > >> At this point send_IPI_allbutself() has been invoked and the system >> just sits and waits on CPU1 to run the function. But, CPU1''s >> evtchn_upcall_mask was set (1), so I''m guessing the pendinginterrupt>> is never acknowledged. > > Okay, the good news is that''s the same bug I was able to repro last > week. Turns out that CPU1''s upcall mask is getting weirdly set under> its feet. Since it''s waiting on the big kernel lock, which is held by> CPU0, which is waiting for acknowledgement of an interrupt in CPU1,we> have a deadlock. > > Given the problem is in that one changeset, this can''t be hard to > track down now.Now fixed in our staging tree. sizeof_vcpu_shift in arch/xen/x86_64/xen_entry.S should be 4, not 3. -- Keir _______________________________________________ Xen- devel mailing list Xen- devel@lists.xensource.com http://lists.xensource.com/xen- devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Thanks Kier. With this fix applied, I am able to boot SMP > dom0. The box I am testing on is x86_64 machine with two > hardware threads. However, if I turn on SMT in the Linux > configuration, the kernel takes a fault in early startup (a > NULL pointer reference at find_busiest_group +144). It > appears that the sched domain hierarchy is not correctly set up here. > Looking at the new smpboot.c, is turning on SMT support no > longer valid?I don''t think we want particularly want to expose CPU sibling relationships to guests, at least not ones that don''t have all their vcpu''s pinned. I think a better approach would probably just to have the guest scheduler favour low numbered VCPUs and leave Xen to do the maping to logical CPUs. This would get us most of the benefit of SMT scheduling, the only trick we''d be missing is the reduced migration penalty for moving a process between logical CPUs sharing an L2 cache. If we felt this was a big deal, we''d need some way of reflecting the current relationship via shared info or such like. Eitherway, no good can come of enabling SMT in the guest right now, though we should fix the crash. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks Ian. I agree with your analysis. K. Y>>> "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk> 10/28/05 7:52 pm >>> > Thanks Kier. With this fix applied, I am able to boot SMP > dom0. The box I am testing on is x86_64 machine with two > hardware threads. However, if I turn on SMT in the Linux > configuration, the kernel takes a fault in early startup (a > NULL pointer reference at find_busiest_group +144). It > appears that the sched domain hierarchy is not correctly set uphere.> Looking at the new smpboot.c, is turning on SMT support no > longer valid?I don''t think we want particularly want to expose CPU sibling relationships to guests, at least not ones that don''t have all their vcpu''s pinned. I think a better approach would probably just to have the guest scheduler favour low numbered VCPUs and leave Xen to do the maping to logical CPUs. This would get us most of the benefit of SMT scheduling, the only trick we''d be missing is the reduced migration penalty for moving a process between logical CPUs sharing an L2 cache. If we felt this was a big deal, we''d need some way of reflecting the current relationship via shared info or such like. Eitherway, no good can come of enabling SMT in the guest right now, though we should fix the crash. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks Ian. I agree with your analysis. K. Y>>> "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk> 10/28/05 7:52 pm >>> > Thanks Kier. With this fix applied, I am able to boot SMP > dom0. The box I am testing on is x86_64 machine with two > hardware threads. However, if I turn on SMT in the Linux > configuration, the kernel takes a fault in early startup (a > NULL pointer reference at find_busiest_group +144). It > appears that the sched domain hierarchy is not correctly set uphere.> Looking at the new smpboot.c, is turning on SMT support no > longer valid?I don''t think we want particularly want to expose CPU sibling relationships to guests, at least not ones that don''t have all their vcpu''s pinned. I think a better approach would probably just to have the guest scheduler favour low numbered VCPUs and leave Xen to do the maping to logical CPUs. This would get us most of the benefit of SMT scheduling, the only trick we''d be missing is the reduced migration penalty for moving a process between logical CPUs sharing an L2 cache. If we felt this was a big deal, we''d need some way of reflecting the current relationship via shared info or such like. Eitherway, no good can come of enabling SMT in the guest right now, though we should fix the crash. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel