Xen devel - Oct 2006 - [PATCH][HVM] fix PCI interrupt routing via ACPI

Hi -

  The attached patch gets PCI interrupt routing working with ACPI.
The basic problem was we weren''t specifying the correct _ADR for
the PCI-ISA bridge in the ACPI DSDT.  (It''s the PCI-ISA bridge that
implements PCI interrupt routing.  Its address is device 1, fn 0 on the
parent PCI bus -- see the _ADR change in the patch.)  This makes attempts
to program the PCI interrupt routing (i.e., to assign a particular IRQ 
to one
of the four PCI "LNK" devices) write into the correct
device''s PCI config
space.  (These writes were going into the PCI-host bridge''s config
space
...)

  But that fix alone causes even more grief with ACPI-compliant guests
that use (the I/O) APIC.  What happens is the fixed PRTA PCI routing table
returned for APIC mode doesn''t reference the LNK devices (which are
still
defined in ACPI), and so calls their _DIS (disable) methods in ACPI, which
effectively stops PCI devices from interrupting.  There are a couple of 
alternatives
for fixing this.  I think the best one (which happens to be easiest, and 
what I''ve
implemented in this patch) is to *always* return the programmable PRTP
routing table, even in APIC mode.  That way the LNK devices are always used
so we don''t need to detect that they''re not used.

  This also has the advantage of allowing APIC-mode guests to better manage
IRQ conflicts.  Normally (on real hardware) the APIC-mode PRT table maps
the four PCI interrupts to IRQs > 15.  But QEMU is (essentially, for 
now) wired
straight into a virtual PIC -- we can''t raise IRQs > 15, so the old 
APIC-mode
PRT was mapping them to IRQS < 16, subjecting them to the possible IRQ-
sharing that the PIC-mode table allows them to adjust via the LNK devices.

  Oh yeah, the patch also corrects a typo in LNKA''s _CRS method. 
Surely
this
should be referencing PIRA, not PIRB (see the other LNKx''s _CRS
methods).

  This combination of things gets rid of complaints about not finding IRQs
for some PCI devices from several ACPI-compliant guests, including Win2003
and RHEL4-64.  It also seemed to fix some more mysterious hang I was seeing
(on xen-unstable only) when trying to boot certain configs with 
ACPI-compliant
guests.

Dave

P.S. The second attachment is a version of the fix that applies after 
the DSDT
fix from qing.he@intel.com.

P.P.S. Keir -- should we be submitting the patched AML (acpi_dsdt.c) as 
well?
I did, but wasn''t sure of the convention ...




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi David,

I agreed with your fix this _ADR for PCI router issue as we discussed
one week before. I think adding more _PRT slot is fine for more new PCI
devices

Yes, current qemu using apic as  pic IRQ0-15''s one to one mapping. Same
thing for MPS table for non-acpi mode.

Now we cut the apic mode support, always return the pic mode IRQ and
Link, that may be fine, but when we want to expand more apic IRQs, what
are we going to do? Then have to going back if need more IRQs. 
By the way, does the code working on SMP guest OS (when apic=1)?

Thanks,

Winston,

-----Original Message-----

Date: Thu, 12 Oct 2006 14:01:16 -0400
From: David Lively <dlively@virtualiron.com>
Subject: [Xen-devel] [PATCH][HVM] fix PCI interrupt routing via ACPI
To: xen-devel@lists.xensource.com
Message-ID: <452E82EC.8000402@virtualiron.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi -

  The attached patch gets PCI interrupt routing working with ACPI.
The basic problem was we weren''t specifying the correct _ADR for
the PCI-ISA bridge in the ACPI DSDT.  (It''s the PCI-ISA bridge that
implements PCI interrupt routing.  Its address is device 1, fn 0 on the
parent PCI bus -- see the _ADR change in the patch.)  This makes
attempts
to program the PCI interrupt routing (i.e., to assign a particular IRQ 
to one
of the four PCI "LNK" devices) write into the correct
device''s PCI
config
space.  (These writes were going into the PCI-host bridge''s config
space

...)

  But that fix alone causes even more grief with ACPI-compliant guests
that use (the I/O) APIC.  What happens is the fixed PRTA PCI routing
table
returned for APIC mode doesn''t reference the LNK devices (which are
still
defined in ACPI), and so calls their _DIS (disable) methods in ACPI,
which
effectively stops PCI devices from interrupting.  There are a couple of 
alternatives
for fixing this.  I think the best one (which happens to be easiest, and

what I''ve
implemented in this patch) is to *always* return the programmable PRTP
routing table, even in APIC mode.  That way the LNK devices are always
used
so we don''t need to detect that they''re not used.

  This also has the advantage of allowing APIC-mode guests to better
manage
IRQ conflicts.  Normally (on real hardware) the APIC-mode PRT table maps
the four PCI interrupts to IRQs > 15.  But QEMU is (essentially, for 
now) wired
straight into a virtual PIC -- we can''t raise IRQs > 15, so the old 
APIC-mode
PRT was mapping them to IRQS < 16, subjecting them to the possible IRQ-
sharing that the PIC-mode table allows them to adjust via the LNK
devices.

  Oh yeah, the patch also corrects a typo in LNKA''s _CRS method. 
Surely

this
should be referencing PIRA, not PIRB (see the other LNKx''s _CRS
methods).

  This combination of things gets rid of complaints about not finding
IRQs
for some PCI devices from several ACPI-compliant guests, including
Win2003
and RHEL4-64.  It also seemed to fix some more mysterious hang I was
seeing
(on xen-unstable only) when trying to boot certain configs with 
ACPI-compliant
guests.

Dave

P.S. The second attachment is a version of the fix that applies after 
the DSDT
fix from qing.he@intel.com.

P.P.S. Keir -- should we be submitting the patched AML (acpi_dsdt.c) as 
well?
I did, but wasn''t sure of the convention ...

-------------- next part --------------
A non-text attachment was scrubbed...
Name: acpi-bios-pci-rt-fixes.patch
Type: text/x-patch
Size: 49220 bytes
Desc: not available
Url :
http://lists.xensource.com/archives/html/xen-devel/attachments/20061012/
34239df8/acpi-bios-pci-rt-fixes.bin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: acpi-bios-pci-rt-fixes-after-xen-platform-dsdt-fix.patch
Type: text/x-patch
Size: 48537 bytes
Desc: not available
Url :
http://lists.xensource.com/archives/html/xen-devel/attachments/20061012/
34239df8/acpi-bios-pci-rt-fixes-after-xen-platform-dsdt-fix.bin

------------------------------

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel


End of Xen-devel Digest, Vol 20, Issue 93
*****************************************

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 10/13/06, Wang, Winston L <winston.l.wang@intel.com>
wrote:
>
> Now we cut the apic mode support, always return the pic mode IRQ and
> Link, that may be fine, but when we want to expand more apic IRQs, what
> are we going to do? Then have to going back if need more IRQs.

If (when?) the device model is changed to have more than 16 IRQ outputs,
I guess there are a couple alternatives to take advantage of it:
  (a) Go back to statically-mapped IRQs into the larger space for APIC mode
      (requires fixing the problem I''ve avoided here, though that
really
shouldn''t
      be hard ...), OR
  (b) Make the LNKx devices'' _PRS methods include IRQs > 16 for APIC
       mode.  This exposes the expanded set of possible IRQs to the
       guest interrupt routing code.  Trivial to implement.

I''m leaning towards (b), of course, since that''s a simple
extension to what
I''ve already written here.  Does that seem reasonable?

By the way, does the code working on SMP guest OS (when
apic=1)?
>
>
I tested it on RHEL4 U2 64 bit SMP guest, which was definitely using the
APIC.  (The original _ADR fix I ran by you last week actually broke APIC
guests since the LNKx _DIS(able) methods started truly disabling emulated
PCI interrupts once they were writing into the proper PCI config space.
This
is what led me down the path to ignoring the PIC/APIC distinction.) 
It''s
also been (lightly) tested with Win2003 Server with the ACPI HAL (PIC
mode).

Dave


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 13/10/06 17:36, "Dave Lively" <dave.lively@gmail.com> wrote:
>   (b) Make the LNKx devices'' _PRS methods include IRQs > 16 for
APIC
>        mode.  This exposes the expanded set of possible IRQs to the
>        guest interrupt routing code.  Trivial to implement.
There are still only 4 LNK devices though (which PCI devices are striped
across), and they cannot address IRQs > 15. These are both limitations of
PIIX hardware. If we break that emulation, we may as well get rid of the
PIIX PCI-ISA bridge device altogether and statically define everything (not
a bad idea imo).

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

That''s true, if we have more than 4  PCI devices need IRQ, then have to
share, it is doesn''t matter how many PRT package entry you added for
each device in pic mode, only have 4  IRQ  to use.

 

And in APIC mode, IRQ is not routed through south bridge PCI IRQ router
for IOAPIC, those four PCI routers are for 8259 PIC. Current QEMU using
INT1_0/15 to IRQ0/15 one to one mapping, but that does not mean no APIC
mode support, if we have bug we have to fix.

 

I think (b) is a short term solution, if we upgrade south bridge from
PIIX3/4 to newer ICHx, we could have 8 PIC mode PCI IRQ router and 24
IOAPIC IRQ in APIC mode. I think IPF platform already support 24 IOAPIC
IRQs now.

 

I would like David separated the  patch, one for PIC mode PCI IRQ
routing _ADR address fix, this one is good, another one for  cutting PIC
mode support, this one may need more discussion. 

 

Regards,

 

Winston,

________________________________

From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] 
Sent: Friday, October 13, 2006 10:00 AM
To: Dave Lively; Wang, Winston L
Cc: xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] [PATCH][HVM] fix PCI interrupt routing via ACPI

 

On 13/10/06 17:36, "Dave Lively" <dave.lively@gmail.com> wrote:

 (b) Make the LNKx devices'' _PRS methods include IRQs > 16 for APIC
       mode.  This exposes the expanded set of possible IRQs to the
       guest interrupt routing code.  Trivial to implement.


There are still only 4 LNK devices though (which PCI devices are striped
across), and they cannot address IRQs > 15. These are both limitations
of PIIX hardware. If we break that emulation, we may as well get rid of
the PIIX PCI-ISA bridge device altogether and statically define
everything (not a bad idea imo).

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 14/10/06 12:27 am, "Wang, Winston L"
<winston.l.wang@intel.com> wrote:
> That¹s true, if we have more than 4  PCI devices need IRQ, then have to
share,
> it is doesn¹t matter how many PRT package entry you added for each device
in
> pic mode, only have 4  IRQ  to use.
>  
> And in APIC mode, IRQ is not routed through south bridge PCI IRQ router for
> IOAPIC, those four PCI routers are for 8259 PIC. Current QEMU using
INT1_0/15
> to IRQ0/15 one to one mapping, but that does not mean no APIC mode support,
if
> we have bug we have to fix.
>  
> I think (b) is a short term solution, if we upgrade south bridge from
PIIX3/4
> to newer ICHx, we could have 8 PIC mode PCI IRQ router and 24 IOAPIC IRQ in
> APIC mode. I think IPF platform already support 24 IOAPIC  IRQs now.
Or, could we simply:
 1. throw away the PCI-ISA bridge device (from qemu and from the ACPI DSDT)
 2. throw away the legacy Œ$PIR¹ routing table in rombios.c
 3. initialise the IRQ field of each PCI device config space to sane unique
values
 4. Make ACPI PRTP static and match the values chosen in (3).
 5. Also throw away all the LNK stuff from ACPI DSDT

If we¹re allowed to do this (programmable PCI-ISA bridge isn¹t part of MP or
ACPI specs, for example) then this seems to involve throwing away a load of
tedious code and thus removing an IRQ-mapping bottleneck. No drawbacks,
unless I¹m ignorant of some PC architectural detail (quite possible)?

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I like the simplification.

What about PIC mode support?  OSes expecting just a PIC may
rely on being able to choose the 4 PCI IRQs.  But I don''t know
this for sure.  In fact, I remember Win2003 with the ACPI PIC HAL
was successfully working around this problem (by leaving the PCI
IRQ assignments unchanged), though it complained in the event
log.  I''m not sure whether we can rely on this or not ...

Also, I worry in general about deviating too much from emulation
of existing chipsets (at least for anything required at guest OS
install time).  Surprising a guest with (perfectly legitimate) combinations
of hardware it hasn''t seen before can be dangerous.

Dave



On 10/14/06, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:
> Or, could we simply:
>  1. throw away the PCI-ISA bridge device (from qemu and from the ACPI
> DSDT)
>  2. throw away the legacy ''$PIR'' routing table in
rombios.c
>  3. initialise the IRQ field of each PCI device config space to sane
> unique values
>  4. Make ACPI PRTP static and match the values chosen in (3).
>  5. Also throw away all the LNK stuff from ACPI DSDT
>
> If we''re allowed to do this (programmable PCI-ISA bridge
isn''t part of MP
> or ACPI specs, for example) then this seems to involve throwing away a load
> of tedious code and thus removing an IRQ-mapping bottleneck. No drawbacks,
> unless I''m ignorant of some PC architectural detail (quite
possible)?
>
>  -- Keir
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 10/13/06, Wang, Winston L <winston.l.wang@intel.com>
wrote:
>
> I think (b) is a short term solution, if we upgrade south bridge from
> PIIX3/4 to newer ICHx, we could have 8 PIC mode PCI IRQ router and 24
IOAPIC
> IRQ in APIC mode. I think IPF platform already support 24 IOAPIC  IRQs now.
>
This may be the safer route, since we know guest OSes are accustomed to ICHx
these days.

I would like David separated the  patch, one for PIC mode PCI IRQ
routing
> _ADR address fix, this one is good, another one for  cutting PIC  mode
> support, this one may need more discussion.
>
Unfortunately, applying only the _ADR fix will break your existing
ACPI-compliant
guests (more than they were broken before).  The LNKx devices are defined in
the
ACPI tables regardless of PIC/APIC mode.  In APIC mode, because the PRTA
routing table returned from _PRT does *not* refer to the LNKx devices, the
guest OS
will disable each LNKx device by calling its _DIS method.  Because the _DIS
method''s
write of PIRx is now routed to the right place (PIIX3 config space - due to
the _ADR fix),
this write now really disables the PCI INTx, preventing any interrupts from
the PCI INTx
lines.

Without the _ADR fix, ACPI guests are finding they can''t redirect the
PCI
LNKx interrupts,
and nicely falling back to using the existing IRQ assignment.  (So
they''re
broken, but in a
way that many guests seem to be able to work around.)

Dave


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 16/10/06 15:51, "Dave Lively" <dave.lively@gmail.com> wrote:
> What about PIC mode support?  OSes expecting just a PIC may
> rely on being able to choose the 4 PCI IRQs.  But I don''t know
> this for sure.  In fact, I remember Win2003 with the ACPI PIC HAL
> was successfully working around this problem (by leaving the PCI
> IRQ assignments unchanged), though it complained in the event
> log.  I''m not sure whether we can rely on this or not ...
I think it¹s unlikely they¹d rely on it. If we don¹t care about the PIC-mode
limitation we could keep the PIIX3 emulation (but I¹m not sure that most
OSes would bother to switch to APIC mode if they have only one processor).
But bothering to try to emulate better PCI-ISA bridge hardware feels like
wasted effort if we can sidestep the entire problem.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 10/16/06, Keir Fraser <Keir.Fraser@cl.cam.ac.uk>
wrote:
>
>  On 16/10/06 15:51, "Dave Lively" <dave.lively@gmail.com>
wrote:
>
>
> What about PIC mode support?  OSes expecting just a PIC may
>  rely on being able to choose the 4 PCI IRQs.  But I don''t know
>  this for sure.  In fact, I remember Win2003 with the ACPI PIC HAL
>  was successfully working around this problem (by leaving the PCI
>  IRQ assignments unchanged), though it complained in the event
>  log.  I''m not sure whether we can rely on this or not ...
>
>  I think it''s unlikely they''d rely on it. If we
don''t care about the PIC-mode limitation
> we could keep the PIIX3 emulation
Are you proposing we use the (current) PIIX3 interrupt routing emulation for
PIC mode guests, and simply hide the interrupt routing functionality for APIC
mode?  That would certainly address my concerns about PIC mode (but note
neither of us is convinced my concerns here are legitimate).
> (but I''m not sure that most OSes would bother to switch to APIC
mode if they have only
Mostly I see SMP kernels using it on uniprocessor configs.
> one processor). But bothering to try to emulate better PCI-ISA bridge
hardware feels like
> wasted effort if we can sidestep the entire problem.
I''m all for sidestepping the problem when that works.  But I worry
that the requirement
of supporting PIC mode means we need this general functionality (i.e.,
PCI interrupt
routing) anyway.  Since there will be no performance advantage in
hiding this functionality
in APIC mode, perhaps we''re better off just leaving it there.

Dave

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 17/10/06 14:26, "Dave Lively" <dave.lively@gmail.com> wrote:
>> one processor). But bothering to try to emulate better PCI-ISA bridge
>> hardware feels like
>> wasted effort if we can sidestep the entire problem.
> 
> I''m all for sidestepping the problem when that works.  But I worry
> that the requirement
> of supporting PIC mode means we need this general functionality (i.e.,
> PCI interrupt
> routing) anyway.  Since there will be no performance advantage in
> hiding this functionality
> in APIC mode, perhaps we''re better off just leaving it there.
I don''t believe that PIC mode requires us to support interrupt routing.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 10/17/06, Keir Fraser <Keir.Fraser@cl.cam.ac.uk>
wrote:
> On 17/10/06 14:26, "Dave Lively" <dave.lively@gmail.com>
wrote:
> > I''m all for sidestepping the problem when that works.  But I
worry
> > that the requirement
> > of supporting PIC mode means we need this general functionality (i.e.,
> > PCI interrupt
> > routing) anyway.  Since there will be no performance advantage in
> > hiding this functionality
> > in APIC mode, perhaps we''re better off just leaving it there.
>
> I don''t believe that PIC mode requires us to support interrupt
routing.
>
>  -- Keir
Sorry, I shouldn''t have said "requirement" ...  I agree
it''s not known
to be necessary (and suspect it''s truly not).

Would you agree it''s *desirable* for PIC mode guests?  Otherwise they
have to live with the default PCI IRQ assignments, which may lead to
unnecessary IRQ sharing.  But I''m not sure how much of a problem this
is in practice (i.e., since the QEMU configs we use are constrained
anyway).

If we didn''t already have a working PCI interrupt router, I
wouldn''t
be proposing that
we implement one (without further evidence that it helps performance).
 But given that
we have one already that works, we might as well use it for PIC mode
guests, at least.

APIC mode guests are a different question.  Once QEMU can raise IRQs >
15, we can either extend the interrupt router to assign IRQs > 15, or
we can hide the interrupt router entirely as you suggest.  I still
suspect the former is easier.  But we don''t need to resolve this issue
now.

Dave

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel