Jan Beulich
2010-Dec-10 09:12 UTC
[Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
Why wouldn''t iommu=0 imply not using the x2apic, if use of the iommu indeed is a prerequisite (which by itself seems questionable as long as APIC IDs aren''t wider than 8 bits)? Looking further at enable_bsp_x2apic(), I also wonder whether the pre-enabled case, when specifying iommu=0 and x2apic=0, can actually work if iommu=0 is a prerequisite (intremap_enabled() would return 1 in that case afaics, so the early check doesn''t help). Thanks, Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Dec-10 10:06 UTC
Re: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
Are you looking at xen-4.0? I think you should look at latest xen-unstable and we can backpoprt patches if that is more satisfactory. I did a big cleanup patch yesterday which makes the code smaller and clearer than it was, yet I don''t understand the dependencies between x2apic-enabled-by-bios and the need for interrupt remapping, and all that stuff. -- Keir On 10/12/2010 09:12, "Jan Beulich" <JBeulich@novell.com> wrote:> Why wouldn''t iommu=0 imply not using the x2apic, if use of the > iommu indeed is a prerequisite (which by itself seems questionable > as long as APIC IDs aren''t wider than 8 bits)? > > Looking further at enable_bsp_x2apic(), I also wonder whether > the pre-enabled case, when specifying iommu=0 and x2apic=0, > can actually work if iommu=0 is a prerequisite (intremap_enabled() > would return 1 in that case afaics, so the early check doesn''t help). > > Thanks, Jan > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2010-Dec-10 10:58 UTC
Re: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
>>> On 10.12.10 at 11:06, Keir Fraser <keir@xen.org> wrote: > Are you looking at xen-4.0? I think you should look at latest xen-unstableI looked at 4.0 in parallel with -unstable (non-staging) as of yesterday.> and we can backpoprt patches if that is more satisfactory. I did a big > cleanup patch yesterday which makes the code smaller and clearer than itI''ll have a look at what you did...> was, yet I don''t understand the dependencies between x2apic-enabled-by-bios > and the need for interrupt remapping, and all that stuff.... after hopefully those dependencies got clarified by one of the original authors. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Dec-10 11:47 UTC
Re: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
On 10/12/2010 10:58, "Jan Beulich" <JBeulich@novell.com> wrote:>>>> On 10.12.10 at 11:06, Keir Fraser <keir@xen.org> wrote: >> Are you looking at xen-4.0? I think you should look at latest xen-unstable > > I looked at 4.0 in parallel with -unstable (non-staging) as of > yesterday.Even before yesterday, xen-unstable benefits from c/s 22388, which is intended to work around the iommu=0/x2apic=0 dependency a bit. Not sure if it is fully satisfactory. And it is not backported to 4.0 as yet because, well, I don''t understand this crap enough, basically.>> and we can backpoprt patches if that is more satisfactory. I did a big >> cleanup patch yesterday which makes the code smaller and clearer than it > > I''ll have a look at what you did...My change (c/s 22475) is larger, but should have no semantic change. It just tries to make things smaller and cleaner. For that alone maybe we will backport it to 4.0 too.>> was, yet I don''t understand the dependencies between x2apic-enabled-by-bios >> and the need for interrupt remapping, and all that stuff. > > ... after hopefully those dependencies got clarified by one of the > original authors.That would be nice. It feels like if BIOS has enabled x2apic we ought to be able to soldier on with it and not need to panic. Couldn''t we ignore the x2apic-ness? Or turn it off? What about AMD boxes that support x2apic on the CPUs yet of course do not have VT-d irq remapping? Lots of questions, few answers. :-) -- Keir> Jan >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2010-Dec-10 12:02 UTC
Re: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
>>> On 10.12.10 at 12:47, Keir Fraser <keir@xen.org> wrote: > On 10/12/2010 10:58, "Jan Beulich" <JBeulich@novell.com> wrote: > >>>>> On 10.12.10 at 11:06, Keir Fraser <keir@xen.org> wrote: >>> Are you looking at xen-4.0? I think you should look at latest xen-unstable >> >> I looked at 4.0 in parallel with -unstable (non-staging) as of >> yesterday. > > Even before yesterday, xen-unstable benefits from c/s 22388, which is > intended to work around the iommu=0/x2apic=0 dependency a bit. Not sure if > it is fully satisfactory. And it is not backported to 4.0 as yet because, > well, I don''t understand this crap enough, basically.Ah, yes, that makes it better (and I must then have looked at an older snapshot of -unstable instead). Still, as you say, a lot of questions remain. On top of what you and I raised already, I wonder whether the APIC-ID-exceeding-8-bits case is really being handled correctly now when iommu=0. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Zhang, Yang Z
2010-Dec-10 15:00 UTC
RE: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
Yes, X2apic is depend on iommu. If disable iommu when using x2apic, it will trigger the panic. best regards yang> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Jan Beulich > Sent: Friday, December 10, 2010 8:03 PM > To: Keir Fraser > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to > using x2apic > > >>> On 10.12.10 at 12:47, Keir Fraser <keir@xen.org> wrote: > > On 10/12/2010 10:58, "Jan Beulich" <JBeulich@novell.com> wrote: > > > >>>>> On 10.12.10 at 11:06, Keir Fraser <keir@xen.org> wrote: > >>> Are you looking at xen-4.0? I think you should look at latest xen-unstable > >> > >> I looked at 4.0 in parallel with -unstable (non-staging) as of > >> yesterday. > > > > Even before yesterday, xen-unstable benefits from c/s 22388, which is > > intended to work around the iommu=0/x2apic=0 dependency a bit. Not sure if > > it is fully satisfactory. And it is not backported to 4.0 as yet because, > > well, I don''t understand this crap enough, basically. > > Ah, yes, that makes it better (and I must then have looked at an > older snapshot of -unstable instead). > > Still, as you say, a lot of questions remain. On top of what you and > I raised already, I wonder whether the APIC-ID-exceeding-8-bits > case is really being handled correctly now when iommu=0. > > Jan > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2010-Dec-10 15:39 UTC
RE: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
>>> On 10.12.10 at 16:00, "Zhang, Yang Z" <yang.z.zhang@intel.com> wrote: > Yes, X2apic is depend on iommu. If disable iommu when using x2apic, it will > trigger the panic.Would mind clarifying where this dependency actually is? Thanks, Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Kay, Allen M
2010-Dec-10 18:26 UTC
RE: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
The architectural requirement is actually between interrupt remapping and
x2apic. Since interrupt remapping is part of the VT-d feature so current
software requires all VT-d features enabled in order for x2apic to be enabled.
Strictly speaking DMA remapping is not required for x2apic. However, queued
invalidation is required since interrupt remapping requires queued invalidation.
So x2apic dependency is as follows:
x2apic->interrupt remapping->queued invalidation
Due to historical reasons, the new VT-d features were built on top of the old
ones as they become available. Is there a requirement to separate this out? If
so, we will need to re-design iommu boot parameter which took a while to get it
right so most systems can now boot successfully.
Allen
-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Jan Beulich
Sent: Friday, December 10, 2010 7:40 AM
To: Zhang, Yang Z
Cc: Keir Fraser; xen-devel@lists.xensource.com; Han, Weidong
Subject: RE: [Xen-devel] iommu=0 leading to panic when system defaults to using
x2apic
>>> On 10.12.10 at 16:00, "Zhang, Yang Z"
<yang.z.zhang@intel.com> wrote:
> Yes, X2apic is depend on iommu. If disable iommu when using x2apic, it will
> trigger the panic.
Would mind clarifying where this dependency actually is?
Thanks, Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Keir Fraser
2010-Dec-10 18:49 UTC
Re: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
Ah, and the interrupt remapping dependency is because PCI(e) devices cannot address 32-bit APIC IDs? -- Keir On 10/12/2010 18:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote:> The architectural requirement is actually between interrupt remapping and > x2apic. Since interrupt remapping is part of the VT-d feature so current > software requires all VT-d features enabled in order for x2apic to be enabled. > > Strictly speaking DMA remapping is not required for x2apic. However, queued > invalidation is required since interrupt remapping requires queued > invalidation. So x2apic dependency is as follows: > > x2apic->interrupt remapping->queued invalidation > > Due to historical reasons, the new VT-d features were built on top of the old > ones as they become available. Is there a requirement to separate this out? > If so, we will need to re-design iommu boot parameter which took a while to > get it right so most systems can now boot successfully. > > Allen > > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Jan Beulich > Sent: Friday, December 10, 2010 7:40 AM > To: Zhang, Yang Z > Cc: Keir Fraser; xen-devel@lists.xensource.com; Han, Weidong > Subject: RE: [Xen-devel] iommu=0 leading to panic when system defaults to > using x2apic > >>>> On 10.12.10 at 16:00, "Zhang, Yang Z" <yang.z.zhang@intel.com> wrote: >> Yes, X2apic is depend on iommu. If disable iommu when using x2apic, it will >> trigger the panic. > > Would mind clarifying where this dependency actually is? > > Thanks, Jan > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Kay, Allen M
2010-Dec-11 00:07 UTC
RE: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
Yes, interrupt remapping is needed to be the intermediary between legacy IOxAPIC and MSI devices and the new x2APIC in the CPU. -----Original Message----- From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser Sent: Friday, December 10, 2010 10:50 AM To: Kay, Allen M; Jan Beulich; Zhang, Yang Z Cc: xen-devel@lists.xensource.com; Han, Weidong Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic Ah, and the interrupt remapping dependency is because PCI(e) devices cannot address 32-bit APIC IDs? -- Keir On 10/12/2010 18:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote:> The architectural requirement is actually between interrupt remapping and > x2apic. Since interrupt remapping is part of the VT-d feature so current > software requires all VT-d features enabled in order for x2apic to be enabled. > > Strictly speaking DMA remapping is not required for x2apic. However, queued > invalidation is required since interrupt remapping requires queued > invalidation. So x2apic dependency is as follows: > > x2apic->interrupt remapping->queued invalidation > > Due to historical reasons, the new VT-d features were built on top of the old > ones as they become available. Is there a requirement to separate this out? > If so, we will need to re-design iommu boot parameter which took a while to > get it right so most systems can now boot successfully. > > Allen > > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Jan Beulich > Sent: Friday, December 10, 2010 7:40 AM > To: Zhang, Yang Z > Cc: Keir Fraser; xen-devel@lists.xensource.com; Han, Weidong > Subject: RE: [Xen-devel] iommu=0 leading to panic when system defaults to > using x2apic > >>>> On 10.12.10 at 16:00, "Zhang, Yang Z" <yang.z.zhang@intel.com> wrote: >> Yes, X2apic is depend on iommu. If disable iommu when using x2apic, it will >> trigger the panic. > > Would mind clarifying where this dependency actually is? > > Thanks, Jan > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2010-Dec-13 08:15 UTC
RE: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
>>> On 11.12.10 at 01:07, "Kay, Allen M" <allen.m.kay@intel.com> wrote: > Yes, interrupt remapping is needed to be the intermediary between legacy > IOxAPIC and MSI devices and the new x2APIC in the CPU.But isn''t this only when there are APIC IDs beyond 255? Jan> -----Original Message----- > From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser > Sent: Friday, December 10, 2010 10:50 AM > To: Kay, Allen M; Jan Beulich; Zhang, Yang Z > Cc: xen-devel@lists.xensource.com; Han, Weidong > Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to > using x2apic > > Ah, and the interrupt remapping dependency is because PCI(e) devices cannot > address 32-bit APIC IDs? > > -- Keir > > On 10/12/2010 18:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote: > >> The architectural requirement is actually between interrupt remapping and >> x2apic. Since interrupt remapping is part of the VT-d feature so current >> software requires all VT-d features enabled in order for x2apic to be > enabled. >> >> Strictly speaking DMA remapping is not required for x2apic. However, queued >> invalidation is required since interrupt remapping requires queued >> invalidation. So x2apic dependency is as follows: >> >> x2apic->interrupt remapping->queued invalidation >> >> Due to historical reasons, the new VT-d features were built on top of the old >> ones as they become available. Is there a requirement to separate this out? >> If so, we will need to re-design iommu boot parameter which took a while to >> get it right so most systems can now boot successfully. >> >> Allen_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Dec-13 09:03 UTC
Re: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
On 13/12/2010 08:15, "Jan Beulich" <JBeulich@novell.com> wrote:>>>> On 11.12.10 at 01:07, "Kay, Allen M" <allen.m.kay@intel.com> wrote: >> Yes, interrupt remapping is needed to be the intermediary between legacy >> IOxAPIC and MSI devices and the new x2APIC in the CPU. > > But isn''t this only when there are APIC IDs beyond 255?Apparently not, since even Linux requires irq remapping even when none of the APIC IDs are greater than 255. Unless running on kvm or xen. I don''t fully understand this particular restriction, mind you. Actually, my guess is that x2apic mode requires a different format of APIC message with a 32-bit APICID field, legacy IOxAPIC and MSI devices do not support the new message format, and so irq remapping hardware is required to bridge the two formats, even if no actual irq remapping is occurring. Is that a canny guess, Allen? -- Keir> Jan > >> -----Original Message----- >> From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser >> Sent: Friday, December 10, 2010 10:50 AM >> To: Kay, Allen M; Jan Beulich; Zhang, Yang Z >> Cc: xen-devel@lists.xensource.com; Han, Weidong >> Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to >> using x2apic >> >> Ah, and the interrupt remapping dependency is because PCI(e) devices cannot >> address 32-bit APIC IDs? >> >> -- Keir >> >> On 10/12/2010 18:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote: >> >>> The architectural requirement is actually between interrupt remapping and >>> x2apic. Since interrupt remapping is part of the VT-d feature so current >>> software requires all VT-d features enabled in order for x2apic to be >> enabled. >>> >>> Strictly speaking DMA remapping is not required for x2apic. However, queued >>> invalidation is required since interrupt remapping requires queued >>> invalidation. So x2apic dependency is as follows: >>> >>> x2apic->interrupt remapping->queued invalidation >>> >>> Due to historical reasons, the new VT-d features were built on top of the >>> old >>> ones as they become available. Is there a requirement to separate this out? >>> If so, we will need to re-design iommu boot parameter which took a while to >>> get it right so most systems can now boot successfully. >>> >>> Allen > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Kay, Allen M
2010-Dec-14 02:25 UTC
RE: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
Keir/Jan, My understanding is that cluster mode requires it. I will get back to you guys after I dig out the details on this - did not get a chance to do this today. Allen -----Original Message----- From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser Sent: Monday, December 13, 2010 1:03 AM To: Jan Beulich; Kay, Allen M; Zhang, Yang Z Cc: Han, Weidong; xen-devel@lists.xensource.com Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic On 13/12/2010 08:15, "Jan Beulich" <JBeulich@novell.com> wrote:>>>> On 11.12.10 at 01:07, "Kay, Allen M" <allen.m.kay@intel.com> wrote: >> Yes, interrupt remapping is needed to be the intermediary between legacy >> IOxAPIC and MSI devices and the new x2APIC in the CPU. > > But isn''t this only when there are APIC IDs beyond 255?Apparently not, since even Linux requires irq remapping even when none of the APIC IDs are greater than 255. Unless running on kvm or xen. I don''t fully understand this particular restriction, mind you. Actually, my guess is that x2apic mode requires a different format of APIC message with a 32-bit APICID field, legacy IOxAPIC and MSI devices do not support the new message format, and so irq remapping hardware is required to bridge the two formats, even if no actual irq remapping is occurring. Is that a canny guess, Allen? -- Keir> Jan > >> -----Original Message----- >> From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser >> Sent: Friday, December 10, 2010 10:50 AM >> To: Kay, Allen M; Jan Beulich; Zhang, Yang Z >> Cc: xen-devel@lists.xensource.com; Han, Weidong >> Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to >> using x2apic >> >> Ah, and the interrupt remapping dependency is because PCI(e) devices cannot >> address 32-bit APIC IDs? >> >> -- Keir >> >> On 10/12/2010 18:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote: >> >>> The architectural requirement is actually between interrupt remapping and >>> x2apic. Since interrupt remapping is part of the VT-d feature so current >>> software requires all VT-d features enabled in order for x2apic to be >> enabled. >>> >>> Strictly speaking DMA remapping is not required for x2apic. However, queued >>> invalidation is required since interrupt remapping requires queued >>> invalidation. So x2apic dependency is as follows: >>> >>> x2apic->interrupt remapping->queued invalidation >>> >>> Due to historical reasons, the new VT-d features were built on top of the >>> old >>> ones as they become available. Is there a requirement to separate this out? >>> If so, we will need to re-design iommu boot parameter which took a while to >>> get it right so most systems can now boot successfully. >>> >>> Allen > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Dec-14 07:44 UTC
Re: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
Well, if it is a restriction imposed by cluster mode, you know the next question is obvious: Why do we bother with cluster mode at all? I don''t see that it yields us any advantage over physical mode, and we could use physical mode without interrupt remapping, that would seem to be a big bonus and simplification? Could we just kill our x2apic cluster mode logic? -- Keir On 14/12/2010 02:25, "Kay, Allen M" <allen.m.kay@intel.com> wrote:> Keir/Jan, > > My understanding is that cluster mode requires it. I will get back to you > guys after I dig out the details on this - did not get a chance to do this > today. > > Allen > > -----Original Message----- > From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser > Sent: Monday, December 13, 2010 1:03 AM > To: Jan Beulich; Kay, Allen M; Zhang, Yang Z > Cc: Han, Weidong; xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to > using x2apic > > On 13/12/2010 08:15, "Jan Beulich" <JBeulich@novell.com> wrote: > >>>>> On 11.12.10 at 01:07, "Kay, Allen M" <allen.m.kay@intel.com> wrote: >>> Yes, interrupt remapping is needed to be the intermediary between legacy >>> IOxAPIC and MSI devices and the new x2APIC in the CPU. >> >> But isn''t this only when there are APIC IDs beyond 255? > > Apparently not, since even Linux requires irq remapping even when none of > the APIC IDs are greater than 255. Unless running on kvm or xen. I don''t > fully understand this particular restriction, mind you. > > Actually, my guess is that x2apic mode requires a different format of APIC > message with a 32-bit APICID field, legacy IOxAPIC and MSI devices do not > support the new message format, and so irq remapping hardware is required to > bridge the two formats, even if no actual irq remapping is occurring. > > Is that a canny guess, Allen? > > -- Keir > >> Jan >> >>> -----Original Message----- >>> From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser >>> Sent: Friday, December 10, 2010 10:50 AM >>> To: Kay, Allen M; Jan Beulich; Zhang, Yang Z >>> Cc: xen-devel@lists.xensource.com; Han, Weidong >>> Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to >>> using x2apic >>> >>> Ah, and the interrupt remapping dependency is because PCI(e) devices cannot >>> address 32-bit APIC IDs? >>> >>> -- Keir >>> >>> On 10/12/2010 18:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote: >>> >>>> The architectural requirement is actually between interrupt remapping and >>>> x2apic. Since interrupt remapping is part of the VT-d feature so current >>>> software requires all VT-d features enabled in order for x2apic to be >>> enabled. >>>> >>>> Strictly speaking DMA remapping is not required for x2apic. However, >>>> queued >>>> invalidation is required since interrupt remapping requires queued >>>> invalidation. So x2apic dependency is as follows: >>>> >>>> x2apic->interrupt remapping->queued invalidation >>>> >>>> Due to historical reasons, the new VT-d features were built on top of the >>>> old >>>> ones as they become available. Is there a requirement to separate this >>>> out? >>>> If so, we will need to re-design iommu boot parameter which took a while to >>>> get it right so most systems can now boot successfully. >>>> >>>> Allen >> >> > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Dec-14 07:59 UTC
Re: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
Also, even if we continued to use cluster mode for IPIs (in the hope of devising a more efficient group IPI algorithm in future) that doesn''t stop us from always exposing physical mode to IOAPICs and MSI devices. -- Keir On 14/12/2010 07:44, "Keir Fraser" <keir@xen.org> wrote:> Well, if it is a restriction imposed by cluster mode, you know the next > question is obvious: Why do we bother with cluster mode at all? I don''t see > that it yields us any advantage over physical mode, and we could use > physical mode without interrupt remapping, that would seem to be a big bonus > and simplification? Could we just kill our x2apic cluster mode logic? > > -- Keir > > On 14/12/2010 02:25, "Kay, Allen M" <allen.m.kay@intel.com> wrote: > >> Keir/Jan, >> >> My understanding is that cluster mode requires it. I will get back to you >> guys after I dig out the details on this - did not get a chance to do this >> today. >> >> Allen >> >> -----Original Message----- >> From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser >> Sent: Monday, December 13, 2010 1:03 AM >> To: Jan Beulich; Kay, Allen M; Zhang, Yang Z >> Cc: Han, Weidong; xen-devel@lists.xensource.com >> Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to >> using x2apic >> >> On 13/12/2010 08:15, "Jan Beulich" <JBeulich@novell.com> wrote: >> >>>>>> On 11.12.10 at 01:07, "Kay, Allen M" <allen.m.kay@intel.com> wrote: >>>> Yes, interrupt remapping is needed to be the intermediary between legacy >>>> IOxAPIC and MSI devices and the new x2APIC in the CPU. >>> >>> But isn''t this only when there are APIC IDs beyond 255? >> >> Apparently not, since even Linux requires irq remapping even when none of >> the APIC IDs are greater than 255. Unless running on kvm or xen. I don''t >> fully understand this particular restriction, mind you. >> >> Actually, my guess is that x2apic mode requires a different format of APIC >> message with a 32-bit APICID field, legacy IOxAPIC and MSI devices do not >> support the new message format, and so irq remapping hardware is required to >> bridge the two formats, even if no actual irq remapping is occurring. >> >> Is that a canny guess, Allen? >> >> -- Keir >> >>> Jan >>> >>>> -----Original Message----- >>>> From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser >>>> Sent: Friday, December 10, 2010 10:50 AM >>>> To: Kay, Allen M; Jan Beulich; Zhang, Yang Z >>>> Cc: xen-devel@lists.xensource.com; Han, Weidong >>>> Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to >>>> using x2apic >>>> >>>> Ah, and the interrupt remapping dependency is because PCI(e) devices cannot >>>> address 32-bit APIC IDs? >>>> >>>> -- Keir >>>> >>>> On 10/12/2010 18:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote: >>>> >>>>> The architectural requirement is actually between interrupt remapping and >>>>> x2apic. Since interrupt remapping is part of the VT-d feature so current >>>>> software requires all VT-d features enabled in order for x2apic to be >>>> enabled. >>>>> >>>>> Strictly speaking DMA remapping is not required for x2apic. However, >>>>> queued >>>>> invalidation is required since interrupt remapping requires queued >>>>> invalidation. So x2apic dependency is as follows: >>>>> >>>>> x2apic->interrupt remapping->queued invalidation >>>>> >>>>> Due to historical reasons, the new VT-d features were built on top of the >>>>> old >>>>> ones as they become available. Is there a requirement to separate this >>>>> out? >>>>> If so, we will need to re-design iommu boot parameter which took a while >>>>> to >>>>> get it right so most systems can now boot successfully. >>>>> >>>>> Allen >>> >>> >> >> > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2010-Dec-14 08:21 UTC
RE: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
>>> On 10.12.10 at 19:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote: > The architectural requirement is actually between interrupt remapping and > x2apic. Since interrupt remapping is part of the VT-d feature so current > software requires all VT-d features enabled in order for x2apic to be enabled. > > Strictly speaking DMA remapping is not required for x2apic. However, queued > invalidation is required since interrupt remapping requires queued > invalidation. So x2apic dependency is as follows: > > x2apic->interrupt remapping->queued invalidation > > Due to historical reasons, the new VT-d features were built on top of the old > ones as they become available. Is there a requirement to separate this out? > If so, we will need to re-design iommu boot parameter which took a while to > get it right so most systems can now boot successfully.And why is it that Xen panics when the BIOS pre-enabled x2apic mode without also enabling interrupt re-mapping, while Linux (afaict from looking at 2.6.32 code and output from an affected machine) simply enables interrupt re-mapping in this case? Thanks, Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Weidong Han
2010-Dec-14 08:46 UTC
Re: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
Jan Beulich wrote:>>>> On 10.12.10 at 19:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote: >>>> >> The architectural requirement is actually between interrupt remapping and >> x2apic. Since interrupt remapping is part of the VT-d feature so current >> software requires all VT-d features enabled in order for x2apic to be enabled. >> >> Strictly speaking DMA remapping is not required for x2apic. However, queued >> invalidation is required since interrupt remapping requires queued >> invalidation. So x2apic dependency is as follows: >> >> x2apic->interrupt remapping->queued invalidation >> >> Due to historical reasons, the new VT-d features were built on top of the old >> ones as they become available. Is there a requirement to separate this out? >> If so, we will need to re-design iommu boot parameter which took a while to >> get it right so most systems can now boot successfully. >> > > And why is it that Xen panics when the BIOS pre-enabled x2apic > mode without also enabling interrupt re-mapping, while Linux > (afaict from looking at 2.6.32 code and output from an affected > machine) simply enables interrupt re-mapping in this case? > > Thanks, Jan > >x2apic spec 2.9 section states: The default will be for the BIOS to pass the control to the OS with the local x2APICs in xAPIC mode if all x2APIC IDs reported by CPUID.0BH:EDX are less than 255, and in x2APIC mode if there are any logical processor reporting its x2APIC ID at 255 or greater. If BIOS transitions to Xen with x2APIC enabled, there must be APIC ID > 255, which requires interrupt remapping. So we added code to check such BIOS issue. Regards, Weidong _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2010-Dec-14 09:12 UTC
Re: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
>>> On 14.12.10 at 09:46, Weidong Han <weidong.han@intel.com> wrote: > Jan Beulich wrote: >> And why is it that Xen panics when the BIOS pre-enabled x2apic >> mode without also enabling interrupt re-mapping, while Linux >> (afaict from looking at 2.6.32 code and output from an affected >> machine) simply enables interrupt re-mapping in this case? >> >> Thanks, Jan >> >> > x2apic spec 2.9 section states: The default will be for the BIOS to pass > the control to the OS with the local x2APICs in xAPIC mode if all x2APIC > IDs reported by CPUID.0BH:EDX are less than 255, and in x2APIC mode if > there are any logical processor reporting its x2APIC ID at 255 or greater. > > If BIOS transitions to Xen with x2APIC enabled, there must be APIC ID > > 255, which requires interrupt remapping. So we added code to check such > BIOS issue.So what would you tell a customer of yours reporting native Linux to boot fine on a certain system, but Xen dying early? Also, from a purely technical perspective, it doesn''t seem like interrupt re-mapping is a requirement for the BIOS: With native Linux booting fine, there obviously is no issue. And clearly, as long as the boot CPU has an APIC id fitting into 8 bits, interrupts ought to work fine without re-mapping. When the boot CPU has a wider APIC ID, obviously the BIOS won''t be able to bring up the system (and hence the question of booting whatever OS on top of it would be mute anyway). Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Kay, Allen M
2010-Dec-14 16:53 UTC
RE: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
> And why is it that Xen panics when the BIOS pre-enabled x2apic > mode without also enabling interrupt re-mapping, while Linux > (afaict from looking at 2.6.32 code and output from an affected > machine) simply enables interrupt re-mapping in this case?AFAIK, Linux also requires interrupt remapping for x2apic. I believe Weidong has fixed the panic issue in Xen when x2apic is enabled without enabling interrupt remapping. It should now behave the same way as Linux. Allen -----Original Message----- From: Jan Beulich [mailto:JBeulich@novell.com] Sent: Tuesday, December 14, 2010 12:22 AM To: Kay, Allen M; Zhang, Yang Z Cc: Han, Weidong; xen-devel@lists.xensource.com; Keir Fraser Subject: RE: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic>>> On 10.12.10 at 19:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote: > The architectural requirement is actually between interrupt remapping and > x2apic. Since interrupt remapping is part of the VT-d feature so current > software requires all VT-d features enabled in order for x2apic to be enabled. > > Strictly speaking DMA remapping is not required for x2apic. However, queued > invalidation is required since interrupt remapping requires queued > invalidation. So x2apic dependency is as follows: > > x2apic->interrupt remapping->queued invalidation > > Due to historical reasons, the new VT-d features were built on top of the old > ones as they become available. Is there a requirement to separate this out? > If so, we will need to re-design iommu boot parameter which took a while to > get it right so most systems can now boot successfully.And why is it that Xen panics when the BIOS pre-enabled x2apic mode without also enabling interrupt re-mapping, while Linux (afaict from looking at 2.6.32 code and output from an affected machine) simply enables interrupt re-mapping in this case? Thanks, Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2010-Dec-14 17:06 UTC
RE: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
>>> On 14.12.10 at 17:53, "Kay, Allen M" <allen.m.kay@intel.com> wrote: >> And why is it that Xen panics when the BIOS pre-enabled x2apic >> mode without also enabling interrupt re-mapping, while Linux >> (afaict from looking at 2.6.32 code and output from an affected >> machine) simply enables interrupt re-mapping in this case? > > AFAIK, Linux also requires interrupt remapping for x2apic. I believe > Weidong has fixed the panic issue in Xen when x2apic is enabled without > enabling interrupt remapping. It should now behave the same way as Linux.No - just look at the very first if() in enable_bsp_x2apic(). Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2010-Dec-14 17:08 UTC
RE: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
>>> On 14.12.10 at 17:53, "Kay, Allen M" <allen.m.kay@intel.com> wrote: >> And why is it that Xen panics when the BIOS pre-enabled x2apic >> mode without also enabling interrupt re-mapping, while Linux >> (afaict from looking at 2.6.32 code and output from an affected >> machine) simply enables interrupt re-mapping in this case? > > AFAIK, Linux also requires interrupt remapping for x2apic. I believe > Weidong has fixed the panic issue in Xen when x2apic is enabled without > enabling interrupt remapping. It should now behave the same way as Linux.Oh, and I should have added that I''m currently polishing up patches from Yinghai Lu to address this as well as other problems preventing (certain?) large boxes from booting or functioning correctly. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yinghai Lu
2010-Dec-14 17:52 UTC
Re: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
On 12/14/2010 09:08 AM, Jan Beulich wrote:>>>> On 14.12.10 at 17:53, "Kay, Allen M" <allen.m.kay@intel.com> wrote: >>> And why is it that Xen panics when the BIOS pre-enabled x2apic >>> mode without also enabling interrupt re-mapping, while Linux >>> (afaict from looking at 2.6.32 code and output from an affected >>> machine) simply enables interrupt re-mapping in this case? >> >> AFAIK, Linux also requires interrupt remapping for x2apic. I believe >> Weidong has fixed the panic issue in Xen when x2apic is enabled without >> enabling interrupt remapping. It should now behave the same way as Linux. > > Oh, and I should have added that I''m currently polishing up > patches from Yinghai Lu to address this as well as other > problems preventing (certain?) large boxes from booting or > functioning correctly.Yes, just want to make xen aligned to current Linux kernel behavior. BIOS guys said why Linux non-xen kernel is happy but xen kernel is not. please check attached patches. Our big box with x2apic-preenabled, AND half cpus'' apic id are above 255. They are on top of xen-4.0.1. and first one is from Weidong in unstable. Thanks Yinghai Lu _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Dec-14 18:18 UTC
Re: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
On 14/12/2010 17:52, "Yinghai Lu" <yinghai@kernel.org> wrote:> On 12/14/2010 09:08 AM, Jan Beulich wrote: >>>>> On 14.12.10 at 17:53, "Kay, Allen M" <allen.m.kay@intel.com> wrote: >>>> And why is it that Xen panics when the BIOS pre-enabled x2apic >>>> mode without also enabling interrupt re-mapping, while Linux >>>> (afaict from looking at 2.6.32 code and output from an affected >>>> machine) simply enables interrupt re-mapping in this case? >>> >>> AFAIK, Linux also requires interrupt remapping for x2apic. I believe >>> Weidong has fixed the panic issue in Xen when x2apic is enabled without >>> enabling interrupt remapping. It should now behave the same way as Linux. >> >> Oh, and I should have added that I''m currently polishing up >> patches from Yinghai Lu to address this as well as other >> problems preventing (certain?) large boxes from booting or >> functioning correctly. > > Yes, just want to make xen aligned to current Linux kernel behavior. > BIOS guys said why Linux non-xen kernel is happy but xen kernel is not. > > please check attached patches. Our big box with x2apic-preenabled, AND half > cpus'' apic id are above 255. > > They are on top of xen-4.0.1. and first one is from Weidong in unstable.I assume Jan is going to help with posting these as a regular patch series for inclusion, with separate patches for 4.0 and 4.1 as appropriate. -- Keir> Thanks > > Yinghai Lu >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Kay, Allen M
2010-Dec-15 02:35 UTC
RE: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
OK, forget what I said about cluster mode in previous message. I investigated into this in more detail today and here is what I found: Keir''s interpretation in previous message is basically correct. Currently, both MSI and IOAPIC address register consists of 32 bits. Bits 31:20 are occupied by 0xFEE so there aren''t enough bits to accommodate full 32-bit APIC ID''s. As a result new format is define for x2apic to accommodate full 32-bit APIC ID''s. The problem is legacy MSI and IOAPIC devices still have 8-bit APIC ID''s so interrupt remapping is used to be the intermediary between the legacy MSI devices and ioapics and CPUs in x2APIC mode. Although it is possible to do turn on x2APIC on systems with less than 256 CPU''s without using interrupt remapping, this configuration is not supported by the architecture in order to simplify HW validation. Instead, BIOS is required to start the system in xAPIC mode for systems with less than 256 CPUs. It will turn on x2APIC mode for systems with more than 256 CPU''s. On these systems, Xen will need to enable VT-d/interrupt remapping for x2APIC to work properly. Allen -----Original Message----- From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser Sent: Monday, December 13, 2010 11:45 PM To: Kay, Allen M; Jan Beulich; Zhang, Yang Z Cc: Han, Weidong; xen-devel@lists.xensource.com Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic Well, if it is a restriction imposed by cluster mode, you know the next question is obvious: Why do we bother with cluster mode at all? I don''t see that it yields us any advantage over physical mode, and we could use physical mode without interrupt remapping, that would seem to be a big bonus and simplification? Could we just kill our x2apic cluster mode logic? -- Keir On 14/12/2010 02:25, "Kay, Allen M" <allen.m.kay@intel.com> wrote:> Keir/Jan, > > My understanding is that cluster mode requires it. I will get back to you > guys after I dig out the details on this - did not get a chance to do this > today. > > Allen > > -----Original Message----- > From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser > Sent: Monday, December 13, 2010 1:03 AM > To: Jan Beulich; Kay, Allen M; Zhang, Yang Z > Cc: Han, Weidong; xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to > using x2apic > > On 13/12/2010 08:15, "Jan Beulich" <JBeulich@novell.com> wrote: > >>>>> On 11.12.10 at 01:07, "Kay, Allen M" <allen.m.kay@intel.com> wrote: >>> Yes, interrupt remapping is needed to be the intermediary between legacy >>> IOxAPIC and MSI devices and the new x2APIC in the CPU. >> >> But isn''t this only when there are APIC IDs beyond 255? > > Apparently not, since even Linux requires irq remapping even when none of > the APIC IDs are greater than 255. Unless running on kvm or xen. I don''t > fully understand this particular restriction, mind you. > > Actually, my guess is that x2apic mode requires a different format of APIC > message with a 32-bit APICID field, legacy IOxAPIC and MSI devices do not > support the new message format, and so irq remapping hardware is required to > bridge the two formats, even if no actual irq remapping is occurring. > > Is that a canny guess, Allen? > > -- Keir > >> Jan >> >>> -----Original Message----- >>> From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser >>> Sent: Friday, December 10, 2010 10:50 AM >>> To: Kay, Allen M; Jan Beulich; Zhang, Yang Z >>> Cc: xen-devel@lists.xensource.com; Han, Weidong >>> Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to >>> using x2apic >>> >>> Ah, and the interrupt remapping dependency is because PCI(e) devices cannot >>> address 32-bit APIC IDs? >>> >>> -- Keir >>> >>> On 10/12/2010 18:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote: >>> >>>> The architectural requirement is actually between interrupt remapping and >>>> x2apic. Since interrupt remapping is part of the VT-d feature so current >>>> software requires all VT-d features enabled in order for x2apic to be >>> enabled. >>>> >>>> Strictly speaking DMA remapping is not required for x2apic. However, >>>> queued >>>> invalidation is required since interrupt remapping requires queued >>>> invalidation. So x2apic dependency is as follows: >>>> >>>> x2apic->interrupt remapping->queued invalidation >>>> >>>> Due to historical reasons, the new VT-d features were built on top of the >>>> old >>>> ones as they become available. Is there a requirement to separate this >>>> out? >>>> If so, we will need to re-design iommu boot parameter which took a while to >>>> get it right so most systems can now boot successfully. >>>> >>>> Allen >> >> > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2010-Dec-15 07:53 UTC
Re: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic
>>> On 14.12.10 at 19:18, Keir Fraser <keir@xen.org> wrote: > On 14/12/2010 17:52, "Yinghai Lu" <yinghai@kernel.org> wrote: >> please check attached patches. Our big box with x2apic-preenabled, AND half >> cpus'' apic id are above 255. >> >> They are on top of xen-4.0.1. and first one is from Weidong in unstable. > > I assume Jan is going to help with posting these as a regular patch series > for inclusion, with separate patches for 4.0 and 4.1 as appropriate.Yes indeed. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel