We experienced the domU interrupt/DMA issue before. The symptom is like the
/proc/interrupts/irq# of Tachyon chip (FC controller) keep on incrementing, but
nobody is serving the interrupts. Eventually the count of these unserved
interrupts reach the max. and are forced to be disabled. With Konrad''s
dom0/domU configuration, this issue is resolved now.
-Ray
-----Original Message-----
From: Bruce Edge [mailto:bruce.edge@gmail.com]
Sent: Friday, October 15, 2010 12:01 PM
To: Lin, Ray
Subject: Fwd: Xen 4.1 interrupts not delievered.
On 13/10/2010 08:00, "Sander Eikelenboom" <linux@eikelenboom.it>
wrote:
> Hello Keir,
>
> OK let''s rephrase, in what cases is it logical that the xen serial
> console freezes together with dom0 ?
> For example some deadlock causes cpu0 to stall on a heavily loaded system
..
> I think having the serial console available to dump the machines state
> is quite vital :-(
Oh, there was a fix for serial interrupt routing: xen-unstable:22148 or
xen-4.0-testing:21342. Are you running a more recent hypervisor than that?
The fix prevents serial interrupt from being migrated away from pcpu0, which
will not work as there is no vector allocated for it on other pcpus. This kind
of fits with the bug you''re seeing, which doesn''t manifest if
you leave pcpu0 unloaded (and hence presumably serial interrupt binding prefers
to stay with unloaded pcpu0).
-- Keir
> I have tried the max_cstate=1 together with the latest
> 2.6.32-xen-next-pvops kernel as dom0 kernel (which Ian''s fix to
the event channels).
> But with the compile test it freezes just as fast.
> Will try xen before changesets 20072/20073 now, probably with 2.6.31
> pvops, since 2.6.32 would need a more recent hypervisor.
>
> --
> Sander
>
>
> Wednesday, October 13, 2010, 1:34:58 AM, you wrote:
>
>> On 12/10/2010 18:17, "Konrad Rzeszutek Wilk"
<konrad.wilk@oracle.com> wrote:
>
>>> A couple of that might fix the problems are:
>>>
>>> 1). Ian''s fix to the event channels:
>>>
http://xenbits.xen.org/gitweb?p=people/ianc/linux-2.6.git;a=commit;h
>>> =5d30cb2
>>> a8
>>> 5912ffb5f6556d55472c26801eef2ea
>>> 2). Disable IRQ balancing in Xen (and also in Linux kernel).
"noirqbalance"
>>> 3). Pin domains, but nothing to Domain 0.
>
>> ITYM cpu 0. Not that this should rightly make any difference that I can
see.
>
>> My suspicion would be the per-CPU IDT patches introduced during 4.0
>> development. Or changes to enable deep C-state sleeps by default. One
>> or the other causing lost interrupts. I think the latter can be
>> discounted by
>> max_cstate=1 as a Xen boot parameter. The former would require trying
>> a build of Xen before and after changesets 20072/20073 -- they are
>> the ones that did the heavy lifting to implement per-CPU IDTs.
>
>> -- Keir
>
>>> But it might be worth trying them out?
>
>
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Oct-18 14:59 UTC
Re: [Xen-devel] FW: Xen 4.1 interrupts not delievered.
On Fri, Oct 15, 2010 at 02:57:44PM -0600, Lin, Ray wrote:> > We experienced the domU interrupt/DMA issue before. The symptom is like the /proc/interrupts/irq# of Tachyon chip (FC controller) keep on incrementing, but nobody is serving the interrupts. Eventually the count of these unserved interrupts reach the max. and are forced to be disabled. With Konrad''s dom0/domU configuration, this issue is resolved now.Can you send the old dom0/domU configuration? I am really curious as of why it would work - the configuration for domU was just mostly made minimal to boot a tiny kernel, and the dom0 was based on Ubuntu/Fedora Core 13 and then enabling all of the Xen options. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
The dom0/domU kernel also affect it. dom0 kernel : 2.6.32.18-pvops-stable domU kernel : 2.6.36-rc6-pvops-kpcif-08 -Ray -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] Sent: Monday, October 18, 2010 7:59 AM To: Lin, Ray Cc: bruce.edge@gmail.com; linux@eikelenboom.it; caker@theshore.net; xen-devel@lists.xensource.com; keir@xen.org; Ian.Campbell@eu.citrix.com; m.a.young@durham.ac.uk; jeremy@goop.org; pasik@iki.fi Subject: Re: [Xen-devel] FW: Xen 4.1 interrupts not delievered. On Fri, Oct 15, 2010 at 02:57:44PM -0600, Lin, Ray wrote:> > We experienced the domU interrupt/DMA issue before. The symptom is like the /proc/interrupts/irq# of Tachyon chip (FC controller) keep on incrementing, but nobody is serving the interrupts. Eventually the count of these unserved interrupts reach the max. and are forced to be disabled. With Konrad''s dom0/domU configuration, this issue is resolved now.Can you send the old dom0/domU configuration? I am really curious as of why it would work - the configuration for domU was just mostly made minimal to boot a tiny kernel, and the dom0 was based on Ubuntu/Fedora Core 13 and then enabling all of the Xen options. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, Oct 18, 2010 at 8:18 AM, Lin, Ray <Ray.Lin@lsi.com> wrote:> > The dom0/domU kernel also affect it. > dom0 kernel : 2.6.32.18-pvops-stable > domU kernel : 2.6.36-rc6-pvops-kpcif-08 > > > -Ray > > -----Original Message----- > From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] > Sent: Monday, October 18, 2010 7:59 AM > To: Lin, Ray > Cc: bruce.edge@gmail.com; linux@eikelenboom.it; caker@theshore.net; xen-devel@lists.xensource.com; keir@xen.org; Ian.Campbell@eu.citrix.com; m.a.young@durham.ac.uk; jeremy@goop.org; pasik@iki.fi > Subject: Re: [Xen-devel] FW: Xen 4.1 interrupts not delievered. > > On Fri, Oct 15, 2010 at 02:57:44PM -0600, Lin, Ray wrote: >> >> We experienced the domU interrupt/DMA issue before. The symptom is like the /proc/interrupts/irq# of Tachyon chip (FC controller) keep on incrementing, but nobody is serving the interrupts. Eventually the count of these unserved interrupts reach the max. and are forced to be disabled. With Konrad''s dom0/domU configuration, this issue is resolved now. > > Can you send the old dom0/domU configuration? I am really curious as of why it would work - the configuration for domU was just mostly made minimal to boot a tiny kernel, and the dom0 was based on Ubuntu/Fedora Core 13 and then enabling all of the Xen options. > >More detail on Ray''s kernel labels (These are from my build system)> dom0 kernel : 2.6.32.18-pvops-stableThis is 2.6.32.12 xen/stable, with the attached config (config.stable-2.6.32.x_dom0_debug)> domU kernel : 2.6.36-rc6-pvops-kpcif-08This is Konrad''s stable/xen-pcifront-0.8.1 with the attached config (config.konrad-pcifront_domU_debug) I''ve attached our working dom0 (config.stable-2.6.32.x_dom0_debug) domU (config.konrad-pcifront_domU_debug) as well as our old non-working config that was used for both dom0 and domU kernels (config.debug) Not sure if you wanted that too or not. In the config.debug that I wanted to use for dom0/domU I threw out all drivers I didn''t need, and made all the dom0''ish drivers modules so they would not take up space in the domU runtime. -Bruce _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel