Konrad Rzeszutek Wilk
2010-Oct-12 17:17 UTC
[Xen-devel] Xen 4.1 interrupts not delievered.
Hey folks,
The last month there was a flurry of emails about PCI passthrough devices and
some that in which a high load is put on machine and interrupts are
not being delivered (to Dom0 it looks that way) - and even the cases where
there are no PCI passthrough enabled.
Three things that pop up is that:
1). Xen 3.4.4-rc1-pre does not have this problem (this is Mr. Christopher
email about 3Ware 9690SA).
2). Moving all Domains from physical cpu 0 makes the problem go away
(this is Mr. Sander email about High cpu load for events/0).
3). No fixes yet.
A couple of that might fix the problems are:
1). Ian''s fix to the event channels:
http://xenbits.xen.org/gitweb?p=people/ianc/linux-2.6.git;a=commit;h=5d30cb2a85912ffb5f6556d55472c26801eef2ea
2). Disable IRQ balancing in Xen (and also in Linux kernel).
"noirqbalance"
3). Pin domains, but nothing to Domain 0.
But it might be worth trying them out?
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
On 12/10/2010 18:17, "Konrad Rzeszutek Wilk" <konrad.wilk@oracle.com> wrote:> A couple of that might fix the problems are: > > 1). Ian''s fix to the event channels: > http://xenbits.xen.org/gitweb?p=people/ianc/linux-2.6.git;a=commit;h=5d30cb2a8 > 5912ffb5f6556d55472c26801eef2ea > 2). Disable IRQ balancing in Xen (and also in Linux kernel). "noirqbalance" > 3). Pin domains, but nothing to Domain 0.ITYM cpu 0. Not that this should rightly make any difference that I can see. My suspicion would be the per-CPU IDT patches introduced during 4.0 development. Or changes to enable deep C-state sleeps by default. One or the other causing lost interrupts. I think the latter can be discounted by max_cstate=1 as a Xen boot parameter. The former would require trying a build of Xen before and after changesets 20072/20073 -- they are the ones that did the heavy lifting to implement per-CPU IDTs. -- Keir> But it might be worth trying them out?_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Sander Eikelenboom
2010-Oct-13 07:00 UTC
[Xen-devel] Re: Xen 4.1 interrupts not delievered.
Hello Keir, OK let''s rephrase, in what cases is it logical that the xen serial console freezes together with dom0 ? For example some deadlock causes cpu0 to stall on a heavily loaded system .. I think having the serial console available to dump the machines state is quite vital :-( I have tried the max_cstate=1 together with the latest 2.6.32-xen-next-pvops kernel as dom0 kernel (which Ian''s fix to the event channels). But with the compile test it freezes just as fast. Will try xen before changesets 20072/20073 now, probably with 2.6.31 pvops, since 2.6.32 would need a more recent hypervisor. -- Sander Wednesday, October 13, 2010, 1:34:58 AM, you wrote:> On 12/10/2010 18:17, "Konrad Rzeszutek Wilk" <konrad.wilk@oracle.com> wrote:>> A couple of that might fix the problems are: >> >> 1). Ian''s fix to the event channels: >> http://xenbits.xen.org/gitweb?p=people/ianc/linux-2.6.git;a=commit;h=5d30cb2a8 >> 5912ffb5f6556d55472c26801eef2ea >> 2). Disable IRQ balancing in Xen (and also in Linux kernel). "noirqbalance" >> 3). Pin domains, but nothing to Domain 0.> ITYM cpu 0. Not that this should rightly make any difference that I can see.> My suspicion would be the per-CPU IDT patches introduced during 4.0 > development. Or changes to enable deep C-state sleeps by default. One or the > other causing lost interrupts. I think the latter can be discounted by > max_cstate=1 as a Xen boot parameter. The former would require trying a > build of Xen before and after changesets 20072/20073 -- they are the ones > that did the heavy lifting to implement per-CPU IDTs.> -- Keir>> But it might be worth trying them out?-- Best regards, Sander mailto:linux@eikelenboom.it _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 13/10/2010 08:00, "Sander Eikelenboom" <linux@eikelenboom.it> wrote:> Hello Keir, > > OK let''s rephrase, in what cases is it logical that the xen serial console > freezes together with dom0 ? > For example some deadlock causes cpu0 to stall on a heavily loaded system .. > I think having the serial console available to dump the machines state is > quite vital :-(Oh, there was a fix for serial interrupt routing: xen-unstable:22148 or xen-4.0-testing:21342. Are you running a more recent hypervisor than that? The fix prevents serial interrupt from being migrated away from pcpu0, which will not work as there is no vector allocated for it on other pcpus. This kind of fits with the bug you''re seeing, which doesn''t manifest if you leave pcpu0 unloaded (and hence presumably serial interrupt binding prefers to stay with unloaded pcpu0). -- Keir> I have tried the max_cstate=1 together with the latest 2.6.32-xen-next-pvops > kernel as dom0 kernel (which Ian''s fix to the event channels). > But with the compile test it freezes just as fast. > Will try xen before changesets 20072/20073 now, probably with 2.6.31 pvops, > since 2.6.32 would need a more recent hypervisor. > > -- > Sander > > > Wednesday, October 13, 2010, 1:34:58 AM, you wrote: > >> On 12/10/2010 18:17, "Konrad Rzeszutek Wilk" <konrad.wilk@oracle.com> wrote: > >>> A couple of that might fix the problems are: >>> >>> 1). Ian''s fix to the event channels: >>> http://xenbits.xen.org/gitweb?p=people/ianc/linux-2.6.git;a=commit;h=5d30cb2 >>> a8 >>> 5912ffb5f6556d55472c26801eef2ea >>> 2). Disable IRQ balancing in Xen (and also in Linux kernel). "noirqbalance" >>> 3). Pin domains, but nothing to Domain 0. > >> ITYM cpu 0. Not that this should rightly make any difference that I can see. > >> My suspicion would be the per-CPU IDT patches introduced during 4.0 >> development. Or changes to enable deep C-state sleeps by default. One or the >> other causing lost interrupts. I think the latter can be discounted by >> max_cstate=1 as a Xen boot parameter. The former would require trying a >> build of Xen before and after changesets 20072/20073 -- they are the ones >> that did the heavy lifting to implement per-CPU IDTs. > >> -- Keir > >>> But it might be worth trying them out? > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel