Xen devel - Jan 2012 - Xen randomly stuck in mdelay() during MP initialization

Hi,

When trying to boot xen 4.1 on new hardware, Xen become stuck in
wakeup_secondary_cpu() in the mdelay function.

    Dprintk("Waiting for send to finish...\n");
    timeout = 0;
    do {
        Dprintk("+");
        udelay(100);
        if ( !x2apic_enabled )
            send_status = apic_read(APIC_ICR) & APIC_ICR_BUSY;
    } while ( send_status && (timeout++ < 1000) );

    printk("before mdelay\n");
    mdelay(10);
    printk("after mdelay\n");

    Dprintk("Deasserting INIT.\n");

The hang can happen randomly with any of the CPUs to wake up and
sometime doesn''t happen at all.
Replacing mdelay(10) with udelay(10) seems to fix the issue.

-- 
Julian

On Wed, Jan 11, 2012 at 01:06:43PM +0000, Julian Pidancet
wrote:
> Hi,
> 
Hello,
> When trying to boot xen 4.1 on new hardware, Xen become stuck in
> wakeup_secondary_cpu() in the mdelay function.
> 
Can you define "new hardware" ?


-- Pasi

On 11/01/2012 13:06, "Julian Pidancet"
<julian.pidancet@gmail.com> wrote:
> Hi,
> 
> When trying to boot xen 4.1 on new hardware, Xen become stuck in
> wakeup_secondary_cpu() in the mdelay function.
> 
>     Dprintk("Waiting for send to finish...\n");
>     timeout = 0;
>     do {
>         Dprintk("+");
>         udelay(100);
>         if ( !x2apic_enabled )
>             send_status = apic_read(APIC_ICR) & APIC_ICR_BUSY;
>     } while ( send_status && (timeout++ < 1000) );
> 
>     printk("before mdelay\n");
>     mdelay(10);
>     printk("after mdelay\n");
> 
>     Dprintk("Deasserting INIT.\n");
> 
> The hang can happen randomly with any of the CPUs to wake up and
> sometime doesn''t happen at all.
> Replacing mdelay(10) with udelay(10) seems to fix the issue.
Do you see this in xen-unstable? Hopefully it is working there, and we can
simply backport the fix.

 -- Keir

On Wed, Jan 11, 2012 at 1:28 PM, Keir Fraser <keir.xen@gmail.com>
wrote:
> On 11/01/2012 13:06, "Julian Pidancet"
<julian.pidancet@gmail.com> wrote:
>
>> Hi,
>>
>> When trying to boot xen 4.1 on new hardware, Xen become stuck in
>> wakeup_secondary_cpu() in the mdelay function.
>>
>>     Dprintk("Waiting for send to finish...\n");
>>     timeout = 0;
>>     do {
>>         Dprintk("+");
>>         udelay(100);
>>         if ( !x2apic_enabled )
>>             send_status = apic_read(APIC_ICR) & APIC_ICR_BUSY;
>>     } while ( send_status && (timeout++ < 1000) );
>>
>>     printk("before mdelay\n");
>>     mdelay(10);
>>     printk("after mdelay\n");
>>
>>     Dprintk("Deasserting INIT.\n");
>>
>> The hang can happen randomly with any of the CPUs to wake up and
>> sometime doesn't happen at all.
>> Replacing mdelay(10) with udelay(10) seems to fix the issue.
>
> Do you see this in xen-unstable? Hopefully it is working there, and we can
> simply backport the fix.
>
I checked yesterday and the code of this function is the same in
xen-unstable, but I don't encounter the problem with xen-unstable.

What concerns me is that this is the only place in the function where
mdelay() is used, the rest of the function seems to be using udelay().
What's the difference between the two ?
Does mdelay() relies on some form of timer to execute ?

-- 
Julian

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 01/11/2012 02:06 PM, Julian Pidancet wrote:
> Hi,
>
> When trying to boot xen 4.1 on new hardware, Xen become stuck in
> wakeup_secondary_cpu() in the mdelay function.
>
>      Dprintk("Waiting for send to finish...\n");
>      timeout = 0;
>      do {
>          Dprintk("+");
>          udelay(100);
>          if ( !x2apic_enabled )
>              send_status = apic_read(APIC_ICR)&  APIC_ICR_BUSY;
>      } while ( send_status&&  (timeout++<  1000) );
>
>      printk("before mdelay\n");
>      mdelay(10);
>      printk("after mdelay\n");
>
>      Dprintk("Deasserting INIT.\n");
>
> The hang can happen randomly with any of the CPUs to wake up and
> sometime doesn''t happen at all.
> Replacing mdelay(10) with udelay(10) seems to fix the issue.
>
Essentially the same issue, check out
  
http://article.gmane.org/gmane.comp.emulators.xen.devel/114776/match=ivy+bridge

Problem is that udelay uses 32 bit value from tsc for measuring
elapsed time and at early boot stage something steals
(most likely SMI) boot cpu with a following wrapping of tsc value
in udelay. And in case of mdelay this happens multiple times.
Replacing mdelay with udelay or removing it helps but it may
break boot on other hardware.
I suspect it''s a BIOS issue.

-- 
Thanks,
  Igor

>>> On 11.01.12 at 15:51, Julian Pidancet
<julian.pidancet@gmail.com> wrote:
> On Wed, Jan 11, 2012 at 1:28 PM, Keir Fraser <keir.xen@gmail.com>
wrote:
>> On 11/01/2012 13:06, "Julian Pidancet"
<julian.pidancet@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> When trying to boot xen 4.1 on new hardware, Xen become stuck in
>>> wakeup_secondary_cpu() in the mdelay function.
>>>
>>>     Dprintk("Waiting for send to finish...\n");
>>>     timeout = 0;
>>>     do {
>>>         Dprintk("+");
>>>         udelay(100);
>>>         if ( !x2apic_enabled )
>>>             send_status = apic_read(APIC_ICR) & APIC_ICR_BUSY;
>>>     } while ( send_status && (timeout++ < 1000) );
>>>
>>>     printk("before mdelay\n");
>>>     mdelay(10);
>>>     printk("after mdelay\n");
>>>
>>>     Dprintk("Deasserting INIT.\n");
>>>
>>> The hang can happen randomly with any of the CPUs to wake up and
>>> sometime doesn''t happen at all.
>>> Replacing mdelay(10) with udelay(10) seems to fix the issue.
>>
>> Do you see this in xen-unstable? Hopefully it is working there, and we
can
>> simply backport the fix.
>>
> 
> I checked yesterday and the code of this function is the same in
> xen-unstable,
Hardly.
> but I don''t encounter the problem with xen-unstable.
Running in x2apic mode, perhaps? (Despite having been asked
already, you still didn''t really provide technical details about your
systems. Nor did you care to attach a log of the successful
attempt with -unstable...)
> What concerns me is that this is the only place in the function where
> mdelay() is used, the rest of the function seems to be using udelay().
> What''s the difference between the two ?
Quite obviously the former delays by a number of milliseconds,
while the value passed to the latter is in microseconds.
> Does mdelay() relies on some form of timer to execute ?
No. mdelay() is simply a loop around udelay() - see
xen/include/xen/delay.h.

Jan

On 11/01/2012 14:51, "Julian Pidancet"
<julian.pidancet@gmail.com> wrote:
> On Wed, Jan 11, 2012 at 1:28 PM, Keir Fraser <keir.xen@gmail.com>
wrote:
>> On 11/01/2012 13:06, "Julian Pidancet"
<julian.pidancet@gmail.com> wrote:
>> 
>>> Hi,
>>> 
>>> When trying to boot xen 4.1 on new hardware, Xen become stuck in
>>> wakeup_secondary_cpu() in the mdelay function.
>>> 
>>>     Dprintk("Waiting for send to finish...\n");
>>>     timeout = 0;
>>>     do {
>>>         Dprintk("+");
>>>         udelay(100);
>>>         if ( !x2apic_enabled )
>>>             send_status = apic_read(APIC_ICR) & APIC_ICR_BUSY;
>>>     } while ( send_status && (timeout++ < 1000) );
>>> 
>>>     printk("before mdelay\n");
>>>     mdelay(10);
>>>     printk("after mdelay\n");
>>> 
>>>     Dprintk("Deasserting INIT.\n");
>>> 
>>> The hang can happen randomly with any of the CPUs to wake up and
>>> sometime doesn''t happen at all.
>>> Replacing mdelay(10) with udelay(10) seems to fix the issue.
>> 
>> Do you see this in xen-unstable? Hopefully it is working there, and we
can
>> simply backport the fix.
>> 
> 
> I checked yesterday and the code of this function is the same in
> xen-unstable, but I don''t encounter the problem with xen-unstable.
Well that would be because the code of this function is not the same in
xen-4.1 and xen-unstable.
> What concerns me is that this is the only place in the function where
> mdelay() is used, the rest of the function seems to be using udelay().
> What''s the difference between the two ?
> Does mdelay() relies on some form of timer to execute ?
mdelay(ms) just calls udelay(1000) ms times.

 -- Keir

On Wed, Jan 11, 2012 at 3:12 PM, Jan Beulich <JBeulich@suse.com>
wrote:
>
> Hardly.
>
>> but I don''t encounter the problem with xen-unstable.
>
> Running in x2apic mode, perhaps? (Despite having been asked
> already, you still didn''t really provide technical details about
your
> systems. Nor did you care to attach a log of the successful
> attempt with -unstable...)
>
Yes, disabling x2apic fixes the issue, thank you.
Is x2apic disabled by default on xen-unstable ? It could explain why
it doesn''t hang with it.
>> Does mdelay() relies on some form of timer to execute ?
>
> No. mdelay() is simply a loop around udelay() - see
> xen/include/xen/delay.h.
>
Yes, that was quite a stupid question from me, what it does is pretty
obvious by looking at the code.

-- 
Julian

>>> On 11.01.12 at 16:28, Julian Pidancet
<julian.pidancet@gmail.com> wrote:
> On Wed, Jan 11, 2012 at 3:12 PM, Jan Beulich <JBeulich@suse.com>
wrote:
>>
>> Hardly.
>>
>>> but I don''t encounter the problem with xen-unstable.
>>
>> Running in x2apic mode, perhaps? (Despite having been asked
>> already, you still didn''t really provide technical details
about your
>> systems. Nor did you care to attach a log of the successful
>> attempt with -unstable...)
>>
> 
> Yes, disabling x2apic fixes the issue, thank you.
> Is x2apic disabled by default on xen-unstable ? It could explain why
> it doesn''t hang with it.
No, it''s enabled by default. But AP bringup was changed for the
x2apic case some time ago (which was the reason for my "Hardly"
above).

Jan