Xen users - Aug 2008 - PCI passthrough - pci interrupt gets disabled

Hi there,


I''m running Xen 3.2.1-rc1-pre on a Ubuntu 8.0.4 server (kernel
2.6.24-19-xen).


I have some network cards which are passed through to the DomUs.
''Normally'' everything works quite well. but sometimes when a
DomU
crashes or has to be shutdown, the pci interrupt of that DomU gets
disabled in Dom0. After that happend there is no way (at least I
haven''t figured out how to do it) to use that network card again in any
DomU or Dom0. Even rebooting the Dom0 does not help. It has to be
turned off and on again to revive that pci interrupt.


root@XenDom:/boot/grub# lspci -nn

00:00.0 Host bridge [0600]: Intel Corporation 5000P Chipset Memory Controller
Hub [8086:25d8] (rev b1)

00:02.0 PCI bridge [0604]: Intel Corporation 5000 Series Chipset PCI Express x8
Port 2-3 [8086:25f7] (rev b1)

00:04.0 PCI bridge [0604]: Intel Corporation 5000 Series Chipset PCI Express x8
Port 4-5 [8086:25f8] (rev b1)

00:06.0 PCI bridge [0604]: Intel Corporation 5000 Series Chipset PCI Express x8
Port 6-7 [8086:25f9] (rev b1)

00:08.0 System peripheral [0880]: Intel Corporation 5000 Series Chipset DMA
Engine [8086:1a38] (rev b1)

00:10.0 Host bridge [0600]: Intel Corporation 5000 Series Chipset FSB Registers
[8086:25f0] (rev b1)

00:10.1 Host bridge [0600]: Intel Corporation 5000 Series Chipset FSB Registers
[8086:25f0] (rev b1)

00:10.2 Host bridge [0600]: Intel Corporation 5000 Series Chipset FSB Registers
[8086:25f0] (rev b1)

00:11.0 Host bridge [0600]: Intel Corporation 5000 Series Chipset Reserved
Registers [8086:25f1] (rev b1)

00:13.0 Host bridge [0600]: Intel Corporation 5000 Series Chipset Reserved
Registers [8086:25f3] (rev b1)

00:15.0 Host bridge [0600]: Intel Corporation 5000 Series Chipset FBD Registers
[8086:25f5] (rev b1)

00:16.0 Host bridge [0600]: Intel Corporation 5000 Series Chipset FBD Registers
[8086:25f6] (rev b1)

00:1c.0 PCI bridge [0604]: Intel Corporation 631xESB/632xESB/3100 Chipset PCI
Express Root Port 1 [8086:2690] (rev 09)

00:1d.0 USB Controller [0c03]: Intel Corporation 631xESB/632xESB/3100 Chipset
UHCI USB Controller #1 [8086:2688] (rev 09)

00:1d.1 USB Controller [0c03]: Intel Corporation 631xESB/632xESB/3100 Chipset
UHCI USB Controller #2 [8086:2689] (rev 09)

00:1d.2 USB Controller [0c03]: Intel Corporation 631xESB/632xESB/3100 Chipset
UHCI USB Controller #3 [8086:268a] (rev 09)

00:1d.7 USB Controller [0c03]: Intel Corporation 631xESB/632xESB/3100 Chipset
EHCI USB2 Controller [8086:268c] (rev 09)

00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev
d9)

00:1f.0 ISA bridge [0601]: Intel Corporation 631xESB/632xESB/3100 Chipset LPC
Interface Controller [8086:2670] (rev 09)

00:1f.1 IDE interface [0101]: Intel Corporation 631xESB/632xESB IDE Controller
[8086:269e] (rev 09)

00:1f.2 IDE interface [0101]: Intel Corporation 631xESB/632xESB/3100 Chipset
SATA IDE Controller [8086:2680] (rev 09)

00:1f.3 SMBus [0c05]: Intel Corporation 631xESB/632xESB/3100 Chipset SMBus
Controller [8086:269b] (rev 09)

01:00.0 PCI bridge [0604]: Intel Corporation 6311ESB/6321ESB PCI Express
Upstream Port [8086:3500] (rev 01)

01:00.3 PCI bridge [0604]: Intel Corporation 6311ESB/6321ESB PCI Express to
PCI-X Bridge [8086:350c] (rev 01)

02:00.0 PCI bridge [0604]: Intel Corporation 6311ESB/6321ESB PCI Express
Downstream Port E1 [8086:3510] (rev 01)

02:02.0 PCI bridge [0604]: Intel Corporation 6311ESB/6321ESB PCI Express
Downstream Port E3 [8086:3518] (rev 01)

03:00.0 PCI bridge [0604]: Intel Corporation 6700PXH PCI Express-to-PCI Bridge A
[8086:0329] (rev 09)

03:00.2 PCI bridge [0604]: Intel Corporation 6700PXH PCI Express-to-PCI Bridge B
[8086:032a] (rev 09)

06:00.0 Ethernet controller [0200]: Intel Corporation 80003ES2LAN Gigabit
Ethernet Controller (Copper) [8086:1096] (rev 01)

06:00.1 Ethernet controller [0200]: Intel Corporation 80003ES2LAN Gigabit
Ethernet Controller (Copper) [8086:1096] (rev 01)

08:00.0 PCI bridge [0604]: Intel Corporation 80333 Segment-A PCI Express-to-PCI
Express Bridge [8086:0370]

08:00.2 PCI bridge [0604]: Intel Corporation 80333 Segment-B PCI Express-to-PCI
Express Bridge [8086:0372]

09:0e.0 RAID bus controller [0104]: Areca Technology Corp. ARC-1220 8-Port
PCI-Express to SATA RAID Controller [17d3:1220]

0c:00.0 PCI bridge [0604]: Integrated Device Technology, Inc. PES12N3A PCI
Express Switch [111d:8018] (rev 04)

0d:00.0 PCI bridge [0604]: Integrated Device Technology, Inc. PES12N3A PCI
Express Switch [111d:8018] (rev 04)

0d:01.0 PCI bridge [0604]: Integrated Device Technology, Inc. PES12N3A PCI
Express Switch [111d:8018] (rev 04)

0e:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet
Controller (Copper) [8086:10bc] (rev 06)

0e:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet
Controller (Copper) [8086:10bc] (rev 06)

0f:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet
Controller (Copper) [8086:10bc] (rev 06)

0f:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet
Controller (Copper) [8086:10bc] (rev 06)

10:01.0 VGA compatible controller [0300]: ATI Technologies Inc ES1000
[1002:515e] (rev 02)


root@XenDom:/boot/grub# dmesg | grep GSI | sort -u

[    0.174640] ACPI: PCI Interrupt 0000:0f:00.1[A] -> GSI 17 (level, low)
-> IRQ 18

[    0.175082] ACPI: PCI Interrupt 0000:0f:00.0[B] -> GSI 18 (level, low)
-> IRQ 17

[    0.175533] ACPI: PCI Interrupt 0000:0e:00.1[A] -> GSI 16 (level, low)
-> IRQ 16

[    0.176047] ACPI: PCI Interrupt 0000:0e:00.0[B] -> GSI 17 (level, low)
-> IRQ 18

[    0.176458] ACPI: PCI Interrupt 0000:06:00.1[B] -> GSI 19 (level, low)
-> IRQ 19

[    0.841866] IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23

[    0.841871] IOAPIC[1]: apic_id 9, version 32, address 0xfec80000, GSI 24-47

[    0.868206] ACPI: PCI Interrupt 0000:06:00.0[A] -> GSI 18 (level, low)
-> IRQ 17

[    0.982087] ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 17 (level, low)
-> IRQ 18

[    1.087619] ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low)
-> IRQ 19

[    1.191635] ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low)
-> IRQ 17

[    1.295767] ACPI: PCI Interrupt 0000:00:1d.7[A] -> GSI 17 (level, low)
-> IRQ 18

[  139.692701] ACPI: PCI Interrupt 0000:06:00.1[B] -> GSI 19 (level, low)
-> IRQ 19

[    2.029267] ACPI: PCI Interrupt 0000:00:02.0[A] -> GSI 16 (level, low)
-> IRQ 16

[    2.029481] ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 16 (level, low)
-> IRQ 16

[    2.029699] ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low)
-> IRQ 16

[    2.029988] ACPI: PCI Interrupt 0000:02:02.0[A] -> GSI 18 (level, low)
-> IRQ 17

[    2.030223] ACPI: PCI Interrupt 0000:00:04.0[A] -> GSI 16 (level, low)
-> IRQ 16

[    2.030496] ACPI: PCI Interrupt 0000:00:06.0[A] -> GSI 16 (level, low)
-> IRQ 16

[    2.030744] ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 17 (level, low)
-> IRQ 18

[    2.907697] ACPI: PCI Interrupt 0000:09:0e.0[A] -> GSI 18 (level, low)
-> IRQ 17

[    3.408474] ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low)
-> IRQ 17

[    3.408868] ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low)
-> IRQ 19

[    3.411717] ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low)
-> IRQ 17

[   37.477182] ACPI: PCI Interrupt 0000:0e:00.0[B] -> GSI 17 (level, low)
-> IRQ 18

[    4.071839] ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low)
-> IRQ 19

[   52.152719] ACPI: PCI Interrupt 0000:0f:00.0[B] -> GSI 18 (level, low)
-> IRQ 17

[   60.809215] ACPI: PCI Interrupt 0000:0f:00.1[A] -> GSI 17 (level, low)
-> IRQ 18

[   69.457842] ACPI: PCI Interrupt 0000:0e:00.1[A] -> GSI 16 (level, low)
-> IRQ 16


When the interrupt is disabled messages.log shows:

...

Aug 14 10:47:42 XenDom kernel: [759772.192543] blkback: ring-ref 9,
event-channel 16, protocol 1 (x86_32-abi)

Aug 14 10:47:42 XenDom kernel: [759772.200813] blkback: ring-ref 10,
event-channel 17, protocol 1 (x86_32-abi)

Aug 14 10:47:42 XenDom kernel: [759772.212134] blkback: ring-ref 11,
event-channel 18, protocol 1 (x86_32-abi)

Aug 14 10:47:43 XenDom kernel: [759773.617958] PCI: Enabling device 0000:06:00.1
(0000 -> 0003)

Aug 14 10:47:43 XenDom kernel: [759773.617983] ACPI: PCI Interrupt
0000:06:00.1[B] -> GSI 19 (level, low) -> IRQ 19

Aug 14 10:50:29 XenDom kernel: [759939.086937] ACPI: PCI interrupt for device
0000:06:00.1 disabled

...


Does anybody know how to prevent DomUs from shutting down shared interrupts?

I''ve tried the noirqdebug kernel option already with no luck.


Thanks for your help, Rado
_________________________________________________________________
Trete mit Deinem inneren Athleten in Verbindung. Beantworte die Fragen!
http://messenger.live.de/mein/mein-messenger-animoticons.aspx

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users