Xen devel - Aug 2007 - dom0, guests, pci hardware and interrupts...

Greetings,

I configured my dom0 to hand a 3com NIC through to a guest:

# Unbind a PCI network card from its network driver
echo -n 0000:04:0d.0 > /sys/bus/pci/drivers/3c59x/unbind
# and now tell PCI Backend to watch the slot.
modprobe pciback
sleep 5
echo -n 0000:04:0d.0 > /sys/bus/pci/drivers/pciback/new_slot
# Now that the backend is watching for the slot, bind to it
echo -n 0000:04:0d.0 > /sys/bus/pci/drivers/pciback/bind

This works fine; the guest sees the NIC after the guest''s py file is
updated
appropriately:
pci =  [ ''04:0d.0'' ]

What happens:

This setup works for awhile.

The interrupt for the NIC is shared with another device used by dom0. In the
best case, dom0 and the guest are running on separate CPUS, and the
interrupt driver in dom0 is called whenever a packet comes in the guest,
which is unnecessary work, imho. In the worst case, when dom0 and the guest
are sharing a CPU, and the dom0 device sharing the interrupt is something
like the disk subsystem, dom0 and the guest appear to steal interrupts from
one another, and dom0 and all the guests grind to a halt (no disk I/O in
this example).

I''ve tried various techniques (kernel parameters in dom0) to ensure
that the
interrupt assigned to the guest''s NIC is *not* shared in order to avoid
this problem, to no avail.

Any suggestions for how to influence the interrupts that are assigned to
devices in dom0? It seems that there are a number of physical interrupts
that are not used, and some are shared:
           CPU0
  1:          2  Phys-irq-level     i8042
  6:          5  Phys-irq-level     floppy
  8:          1  Phys-irq-level     rtc
  9:          0  Phys-irq-level     acpi
 14:      89901  Phys-irq-level     libata
 15:       9347  Phys-irq-level     libata
 16:          0  Phys-irq-level     uhci_hcd:usb1
 17:     682628  Phys-irq-level     uhci_hcd:usb2, peth0
 18:      96245  Phys-irq-level     aic7xxx
 19:        356  Phys-irq-level     Intel 82801BA-ICH2, eth1
256:     263502  Dynamic-irq-level     timer0
257:          0  Dynamic-irq-level     resched0
258:          0  Dynamic-irq-level     callfunc0
259:       3823  Dynamic-irq-level     xenbus
260:        180  Dynamic-irq-level     console
261:      18936  Dynamic-irq-level     blkif-backend
262:       9431  Dynamic-irq-level     blkif-backend
263:      20035  Dynamic-irq-level     blkif-backend
264:      44080  Dynamic-irq-level     vif1.0
265:      12878  Dynamic-irq-level     blkif-backend
266:       1071  Dynamic-irq-level     vif4.0
267:       3352  Dynamic-irq-level     vif2.0
268:      17074  Dynamic-irq-level     blkif-backend
269:       5598  Dynamic-irq-level     vif5.0
270:     118463  Dynamic-irq-level     vif3.0

One would think that if the physical interrupts are available, and the
device can be configured to use them, that no sharing would occur until the
system allocated all the physical interrupts.. 

TIA,
Mike







_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 10/8/07 17:15, "Mike Carney" <mc-al34luc@sbcglobal.net>
wrote:
> Any suggestions for how to influence the interrupts that are assigned to
> devices in dom0? It seems that there are a number of physical interrupts
> that are not used, and some are shared:
If the device interrupts are aliasing then that is because the physical
interrupt lines are physically wired together. There''s nothing you can
do
about that, except try moving PCI cards to different slots (if one of your
devices is not built in to the mobo chipset) and hence use a different
physical interrupt line for that PCI card.

Which interrupt number is shared? If you ''xm debug-key z'' and
then ''xm
dmesg'', a bunch of IO-APIC related debug output should be printed. This
might yield some interesting info (probably just confirming that your
devices really are wired to the same interrupt line).

 -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Keir,

Thanks for your response.

Keir Fraser wrote:
> On 10/8/07 17:15, "Mike Carney" <mc-al34luc@sbcglobal.net>
wrote:
> 
>> Any suggestions for how to influence the interrupts that are assigned
to
>> devices in dom0? It seems that there are a number of physical
interrupts
>> that are not used, and some are shared:
> 
> If the device interrupts are aliasing then that is because the physical
> interrupt lines are physically wired together. There''s nothing you
can do
> about that, except try moving PCI cards to different slots (if one of your
> devices is not built in to the mobo chipset) and hence use a different
> physical interrupt line for that PCI card.
> 
> Which interrupt number is shared? If you ''xm debug-key z''
and then ''xm
> dmesg'', a bunch of IO-APIC related debug output should be printed.
This
> might yield some interesting info (probably just confirming that your
> devices really are wired to the same interrupt line).
> 
>  -- Keir
Attached please find the output of xm dmesg after xm debug-key z. Any
help in parsing this would be appreciated.

Thanks,

Mike




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel