thr3ads.net - Xen devel - [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem [Sep 2010]

If this information is useful, please help other people find it:
Share via:

Bruce Edge

2010-Sep-27 15:52 UTC

[Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

One of our developers who is working on a tachyon driver is
complaining that the pvops domU kernel is not working for these MSI
interrupts.
This is using the current head of xen/2.6.32.x on both a single
Nahelam 920 and a dual E5540. This behavior is consistent with Xen
4.0.1, 4.0.2.rc1-pre and 4.1.

Here are his comments:

- the driver has no problem to enable msi interrupt and request the
interrupt through kernel functions pci_enable_msi & request_irq
- the interrupt does happen. But the interrupt service routine of
tachyon driver doesn''t detect any interrupt status related to this
interrupt, which inhibits the tachyon chip from coming on-line. And
there are high count of tachyon interrupt in /proc/interrupts

kaan-18-dpm:~# cat /proc/interrupts | grep TACH
124:     760415          0          0          0          0          0
         0          0          0          0          0          0
    0          0  xen-pirq-pcifront-msi  HW_TACHYON
125:     762234          0          0          0          0          0
         0          0          0          0          0          0
    0          0  xen-pirq-pcifront-msi  HW_TACHYON
126:     764180          0          0          0          0          0
         0          0          0          0          0          0
    0          0  xen-pirq-pcifront-msi  HW_TACHYON
127:     764164          0          0          0          0          0
         0          0          0          0          0          0
    0          0  xen-pirq-pcifront-msi  HW_TACHYON


-Bruce

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk

2010-Sep-27 17:24 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge
wrote:> One of our developers who is working on a tachyon driver is
> complaining that the pvops domU kernel is not working for these MSI
> interrupts.
> This is using the current head of xen/2.6.32.x on both a single
> Nahelam 920 and a dual E5540. This behavior is consistent with Xen
> 4.0.1, 4.0.2.rc1-pre and 4.1.
> 
> Here are his comments:
> 
> - the driver has no problem to enable msi interrupt and request the
> interrupt through kernel functions pci_enable_msi & request_irq
What shows up in the Xen console when you send the ''q'' key?
Does it
show that the vector is assigned to the appropiate guest?
> - the interrupt does happen. But the interrupt service routine of
> tachyon driver doesn''t detect any interrupt status related to this
> interrupt, which inhibits the tachyon chip from coming on-line. And
> there are high count of tachyon interrupt in /proc/interrupts
Is it checking the PCI_STATUS_INTERRUPT or the appropiate register
in the MMIO BAR?
> 
> kaan-18-dpm:~# cat /proc/interrupts | grep TACH
> 124:     760415          0          0          0          0          0
>          0          0          0          0          0          0
>     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> 125:     762234          0          0          0          0          0
>          0          0          0          0          0          0
>     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> 126:     764180          0          0          0          0          0
>          0          0          0          0          0          0
>     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> 127:     764164          0          0          0          0          0
>          0          0          0          0          0          0
>     0          0  xen-pirq-pcifront-msi  HW_TACHYON
Can you provide the full dmesg output?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Bruce Edge

2010-Sep-27 19:16 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Mon, Sep 27, 2010 at 10:24 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:>
> On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge wrote:
> > One of our developers who is working on a tachyon driver is
> > complaining that the pvops domU kernel is not working for these MSI
> > interrupts.
> > This is using the current head of xen/2.6.32.x on both a single
> > Nahelam 920 and a dual E5540. This behavior is consistent with Xen
> > 4.0.1, 4.0.2.rc1-pre and 4.1.
> >
> > Here are his comments:
> >
> > - the driver has no problem to enable msi interrupt and request the
> > interrupt through kernel functions pci_enable_msi & request_irq
>
> What shows up in the Xen console when you send the ''q''
key? Does it
> show that the vector is assigned to the appropiate guest?
The Xen console q key shows that the domU is assigned:

(XEN)     Interrupts { 32, 41-42, 47 }

but the domU thinks it has:

124/125/126/127

Is there some mapping that''s taking place, or is this plain wrong?
>
> > - the interrupt does happen. But the interrupt service routine of
> > tachyon driver doesn''t detect any interrupt status related to
this
> > interrupt, which inhibits the tachyon chip from coming on-line. And
> > there are high count of tachyon interrupt in /proc/interrupts
>
> Is it checking the PCI_STATUS_INTERRUPT or the appropiate register
> in the MMIO BAR?
>
The driver would check the appropriate register (tachyon registers) in
the MMIO to determine the source of interrupts.
> >
> > kaan-18-dpm:~# cat /proc/interrupts | grep TACH
> > 124:     760415          0          0          0          0          0
> >          0          0          0          0          0          0
> >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> > 125:     762234          0          0          0          0          0
> >          0          0          0          0          0          0
> >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> > 126:     764180          0          0          0          0          0
> >          0          0          0          0          0          0
> >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> > 127:     764164          0          0          0          0          0
> >          0          0          0          0          0          0
> >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
>
> Can you provide the full dmesg output?
Attached.

Some possibly related messages on dom0 console:

[ 1882.269778] pciback 0000:07:00.0: enabling device (0000 -> 0003)
[ 1882.269800] xen: registering gsi 32 triggering 0 polarity 1
[ 1882.269827] xen_allocate_pirq: returning irq 32 for gsi 32
[ 1882.269834] xen: --> irq=32
[ 1882.269841] Already setup the GSI :32
[ 1882.269847] pciback 0000:07:00.0: PCI INT A -> GSI 32 (level, low) ->
IRQ 32
[ 1882.269866] pciback 0000:07:00.0: setting latency timer to 64
[ 1882.270463] pciback 0000:07:00.0: Driver tried to write to a
read-only configuration space field at offset 0x62, size 2. This may
be harmless, but if you have problems with your device:
[ 1882.270465] 1) see permissive attribute in sysfs
[ 1882.270467] 2) report problems to the xen-devel mailing list along
with details of your device obtained from lspci.
[ 1882.270615]   alloc irq_desc for 478 on node 0
[ 1882.270625]   alloc kstat_irqs on node 0
[ 1882.348411] pciback 0000:07:00.1: enabling device (0000 -> 0003)
[ 1882.348433] xen: registering gsi 42 triggering 0 polarity 1
[ 1882.348440] xen_allocate_pirq: returning irq 42 for gsi 42
[ 1882.348445] xen: --> irq=42
[ 1882.348472] Already setup the GSI :42
[ 1882.348479] pciback 0000:07:00.1: PCI INT B -> GSI 42 (level, low) ->
IRQ 42
[ 1882.348497] pciback 0000:07:00.1: setting latency timer to 64
[ 1882.349063] pciback 0000:07:00.1: Driver tried to write to a
read-only configuration space field at offset 0x62, size 2. This may
be harmless, but if you have problems with your device:
[ 1882.349066] 1) see permissive attribute in sysfs
[ 1882.349067] 2) report problems to the xen-devel mailing list along
with details of your device obtained from lspci.
[ 1882.349205]   alloc irq_desc for 477 on node 0
[ 1882.349215]   alloc kstat_irqs on node 0
[ 1882.402893] pciback 0000:07:00.2: enabling device (0000 -> 0003)
[ 1882.402908] xen: registering gsi 47 triggering 0 polarity 1
[ 1882.402913] xen_allocate_pirq: returning irq 47 for gsi 47
[ 1882.402916] xen: --> irq=47
[ 1882.402921] Already setup the GSI :47
[ 1882.402925] pciback 0000:07:00.2: PCI INT C -> GSI 47 (level, low) ->
IRQ 47
[ 1882.402938] pciback 0000:07:00.2: setting latency timer to 64
[ 1882.403280] pciback 0000:07:00.2: Driver tried to write to a
read-only configuration space field at offset 0x62, size 2. This may
be harmless, but if you have problems with your device:
[ 1882.403282] 1) see permissive attribute in sysfs
[ 1882.403282] 2) report problems to the xen-devel mailing list along
with details of your device obtained from lspci.
[ 1882.403380]   alloc irq_desc for 476 on node 0
[ 1882.403386]   alloc kstat_irqs on node 0
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr e6f80000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = e6f80
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 3
(XEN)     l3[3] = 702b6003
(XEN)     l2 = ffff8300702b6000
(XEN)     l2_index = 137
(XEN)     l2[137] = 0
(XEN)     l2[137] not present
(XEN) traps.c:466:d0 Unhandled nmi fault/trap [#2] on VCPU 0 [ec=0000]
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0
(XEN)     l2[6] not present
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0
(XEN)     l2[6] not present
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0
(XEN)     l2[6] not present
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0
(XEN)     l2[6] not present
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0
(XEN)     l2[6] not present
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0
(XEN)     l2[6] not present
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0
(XEN)     l2[6] not present
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0
(XEN)     l2[6] not present
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0
(XEN)     l2[6] not present
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0
(XEN)     l2[6] not present
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0
(XEN)     l2[6] not present
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0
(XEN)     l2[6] not present
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0
(XEN)     l2[6] not present
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0
(XEN)     l2[6] not present
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0
(XEN)     l2[6] not present
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0
(XEN)     l2[6] not present
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0
(XEN)     l2[6] not present
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0
(XEN)     l2[6] not present
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0
(XEN)     l2[6] not present
(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc00




-Bruce


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk

2010-Sep-27 19:54 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Mon, Sep 27, 2010 at 12:16:50PM -0700, Bruce Edge
wrote:> On Mon, Sep 27, 2010 at 10:24 AM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> >
> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge wrote:
> > > One of our developers who is working on a tachyon driver is
> > > complaining that the pvops domU kernel is not working for these
MSI
> > > interrupts.
> > > This is using the current head of xen/2.6.32.x on both a single
> > > Nahelam 920 and a dual E5540. This behavior is consistent with
Xen
> > > 4.0.1, 4.0.2.rc1-pre and 4.1.
> > >
> > > Here are his comments:
> > >
> > > - the driver has no problem to enable msi interrupt and request
the
> > > interrupt through kernel functions pci_enable_msi &
request_irq
> >
> > What shows up in the Xen console when you send the
''q'' key? Does it
> > show that the vector is assigned to the appropiate guest?
> 
> The Xen console q key shows that the domU is assigned:
> 
> (XEN)     Interrupts { 32, 41-42, 47 }
Aha!
> 
> but the domU thinks it has:
> 
> 124/125/126/127
> 
> Is there some mapping that''s taking place, or is this plain wrong?
That looks wrong. The IRQ numbers (even though they are MSI vectors) are
setup as IRQ numbers in the DomU guest. You should have seen

32:
41:
42:
47:
in you /proc/interrupts on your DomU guest.

I wonder what broke  - can you use
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
devel/xen-pcifront-0.5 (or pv/pcifront-2.6.32)?

It has the latest pcifront driver but without the PVonHVM enhancments
so we can try to eliminate the PvONHVM logic out of the picture.
> 
> >
> > > - the interrupt does happen. But the interrupt service routine of
> > > tachyon driver doesn''t detect any interrupt status
related to this
> > > interrupt, which inhibits the tachyon chip from coming on-line.
And
> > > there are high count of tachyon interrupt in /proc/interrupts
> >
> > Is it checking the PCI_STATUS_INTERRUPT or the appropiate register
> > in the MMIO BAR?
> >
> 
> The driver would check the appropriate register (tachyon registers) in
> the MMIO to determine the source of interrupts.
OK, so that isn''t it. Is there anything at these vectors:
7c, 7d, 7e, and 7f? When you use xen debug-keys ''i'' or
''q'' it should give you
an inkling what device this is set for.
> 
> > >
> > > kaan-18-dpm:~# cat /proc/interrupts | grep TACH
> > > 124:     760415          0          0          0          0      
   0
> > >          0          0          0          0          0          0
> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> > > 125:     762234          0          0          0          0      
   0
> > >          0          0          0          0          0          0
> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> > > 126:     764180          0          0          0          0      
   0
> > >          0          0          0          0          0          0
> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> > > 127:     764164          0          0          0          0      
   0
> > >          0          0          0          0          0          0
> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> >
> > Can you provide the full dmesg output?
> 
> Attached.
> 
> Some possibly related messages on dom0 console:
> 
> [ 1882.269778] pciback 0000:07:00.0: enabling device (0000 -> 0003)
> [ 1882.269800] xen: registering gsi 32 triggering 0 polarity 1
> [ 1882.269827] xen_allocate_pirq: returning irq 32 for gsi 32
> [ 1882.269834] xen: --> irq=32
> [ 1882.269841] Already setup the GSI :32
> [ 1882.269847] pciback 0000:07:00.0: PCI INT A -> GSI 32 (level, low)
-> IRQ 32
> [ 1882.269866] pciback 0000:07:00.0: setting latency timer to 64
> [ 1882.270463] pciback 0000:07:00.0: Driver tried to write to a
> read-only configuration space field at offset 0x62, size 2. This may
> be harmless, but if you have problems with your device:
Uhhh, for that I think you need to do ''lspci -vvv -xxx -s
07:00.00''
to find out what is at the configuration space. You could enable
it using the permissive attribute.
> [ 1882.270465] 1) see permissive attribute in sysfs
> [ 1882.270467] 2) report problems to the xen-devel mailing list along
> with details of your device obtained from lspci.
> [ 1882.270615]   alloc irq_desc for 478 on node 0
> [ 1882.270625]   alloc kstat_irqs on node 0
So for 478: what do you see? xen-pciback I presume?> [ 1882.348411] pciback 0000:07:00.1: enabling device (0000 -> 0003)
> [ 1882.348433] xen: registering gsi 42 triggering 0 polarity 1
> [ 1882.348440] xen_allocate_pirq: returning irq 42 for gsi 42
> [ 1882.348445] xen: --> irq=42
> [ 1882.348472] Already setup the GSI :42
> [ 1882.348479] pciback 0000:07:00.1: PCI INT B -> GSI 42 (level, low)
-> IRQ 42
> [ 1882.348497] pciback 0000:07:00.1: setting latency timer to 64
> [ 1882.349063] pciback 0000:07:00.1: Driver tried to write to a
> read-only configuration space field at offset 0x62, size 2. This may
> be harmless, but if you have problems with your device:
> [ 1882.349066] 1) see permissive attribute in sysfs
> [ 1882.349067] 2) report problems to the xen-devel mailing list along
> with details of your device obtained from lspci.
> [ 1882.349205]   alloc irq_desc for 477 on node 0
> [ 1882.349215]   alloc kstat_irqs on node 0
> [ 1882.402893] pciback 0000:07:00.2: enabling device (0000 -> 0003)
> [ 1882.402908] xen: registering gsi 47 triggering 0 polarity 1
> [ 1882.402913] xen_allocate_pirq: returning irq 47 for gsi 47
> [ 1882.402916] xen: --> irq=47
> [ 1882.402921] Already setup the GSI :47
> [ 1882.402925] pciback 0000:07:00.2: PCI INT C -> GSI 47 (level, low)
-> IRQ 47
> [ 1882.402938] pciback 0000:07:00.2: setting latency timer to 64
> [ 1882.403280] pciback 0000:07:00.2: Driver tried to write to a
> read-only configuration space field at offset 0x62, size 2. This may
> be harmless, but if you have problems with your device:
> [ 1882.403282] 1) see permissive attribute in sysfs
> [ 1882.403282] 2) report problems to the xen-devel mailing list along
> with details of your device obtained from lspci.
> [ 1882.403380]   alloc irq_desc for 476 on node 0
> [ 1882.403386]   alloc kstat_irqs on node 0
> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
> fault addr e6f80000, iommu reg = ffff82c3fff57000
> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
> (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = e6f80
> (XEN)     root_entry = ffff83019ff70000
> (XEN)     root_entry[7] = 19cf52001
> (XEN)     context = ffff83019cf52000
> (XEN)     context[0] = 102_706dc005
> (XEN)     l4 = ffff8300706dc000
> (XEN)     l4_index = 0
> (XEN)     l4[0] = 706db003
> (XEN)     l3 = ffff8300706db000
> (XEN)     l3_index = 3
> (XEN)     l3[3] = 702b6003
> (XEN)     l2 = ffff8300702b6000
> (XEN)     l2_index = 137
> (XEN)     l2[137] = 0
> (XEN)     l2[137] not present
> (XEN) traps.c:466:d0 Unhandled nmi fault/trap [#2] on VCPU 0 [ec=0000]
That is not good. What changed from your earlier emails that this was triggered?
Or
was it triggered all along? What happens if you run the system without the iommu
enabled?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Bruce Edge

2010-Sep-27 23:54 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Mon, Sep 27, 2010 at 12:54 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:> On Mon, Sep 27, 2010 at 12:16:50PM -0700, Bruce Edge wrote:
>> On Mon, Sep 27, 2010 at 10:24 AM, Konrad Rzeszutek Wilk
>> <konrad.wilk@oracle.com> wrote:
>> >
>> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge wrote:
>> > > One of our developers who is working on a tachyon driver is
>> > > complaining that the pvops domU kernel is not working for
these MSI
>> > > interrupts.
>> > > This is using the current head of xen/2.6.32.x on both a
single
>> > > Nahelam 920 and a dual E5540. This behavior is consistent
with Xen
>> > > 4.0.1, 4.0.2.rc1-pre and 4.1.
>> > >
>> > > Here are his comments:
>> > >
>> > > - the driver has no problem to enable msi interrupt and
request the
>> > > interrupt through kernel functions pci_enable_msi &
request_irq
>> >
>> > What shows up in the Xen console when you send the
''q'' key? Does it
>> > show that the vector is assigned to the appropiate guest?
>>
>> The Xen console q key shows that the domU is assigned:
>>
>> (XEN)     Interrupts { 32, 41-42, 47 }
>
> Aha!
>
>>
>> but the domU thinks it has:
>>
>> 124/125/126/127
>>
>> Is there some mapping that''s taking place, or is this plain
wrong?
>
> That looks wrong. The IRQ numbers (even though they are MSI vectors) are
> setup as IRQ numbers in the DomU guest. You should have seen
>
> 32:
> 41:
> 42:
> 47:
> in you /proc/interrupts on your DomU guest.
>
> I wonder what broke  - can you use
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
> devel/xen-pcifront-0.5 (or pv/pcifront-2.6.32)?
Please forgive the git ignorance.

Is this the right syntax?

git clone
git://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.32
linux-2.6.32-pv-pcifront

Initialized empty Git repository in
/import/kaan/bedge/src/xen/kernel/pv-ops/linux-2.6.32-pv-pcifront/.git/
fatal: The remote end hung up unexpectedly

Or:

 git clone  git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git

Initialized empty Git repository in
/import/kaan/bedge/src/xen/kernel/pv-ops/xen/.git/
remote: error: Could not read 59eab2f8f04147c5aadc99f2034ca7e5b81e890f
remote: fatal: Failed to traverse parents of commit
979e121cb348add17ed8171bf447b27a3a9d1be3
remote: aborting due to possible repository corruption on the remote side.
fatal: early EOF
fatal: index-pack failed
>
> It has the latest pcifront driver but without the PVonHVM enhancments
> so we can try to eliminate the PvONHVM logic out of the picture.
>
>>
>> >
>> > > - the interrupt does happen. But the interrupt service
routine of
>> > > tachyon driver doesn''t detect any interrupt status
related to this
>> > > interrupt, which inhibits the tachyon chip from coming
on-line. And
>> > > there are high count of tachyon interrupt in /proc/interrupts
>> >
>> > Is it checking the PCI_STATUS_INTERRUPT or the appropiate register
>> > in the MMIO BAR?
>> >
>>
>> The driver would check the appropriate register (tachyon registers) in
>> the MMIO to determine the source of interrupts.
>
> OK, so that isn''t it. Is there anything at these vectors:
> 7c, 7d, 7e, and 7f? When you use xen debug-keys ''i'' or
''q'' it should give you
> an inkling what device this is set for.
When I run a distro kernel in hvm mode, I get the expected irq mappings:

''i'' - Note 66 - 69
(XEN)    IRQ:  66 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:3a
type=PCI-MSI         status=00000010 in-flight=0
domain-list=10:127(----),
(XEN)    IRQ:  67 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:42
type=PCI-MSI         status=00000010 in-flight=0
domain-list=10:126(----),
(XEN)    IRQ:  68 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:4a
type=PCI-MSI         status=00000010 in-flight=0
domain-list=10:125(----),
(XEN)    IRQ:  69 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:52
type=PCI-MSI         status=00000010 in-flight=0
domain-list=10:124(----)


''q''
(XEN)     Interrupts { 32, 41-42, 47, 124-127 }


The same data with pv-ops kernel shows:

''i''
IRQ numbers stop at 65, no 66 - 69 present:

(XEN)    IRQ:  63 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:91
type=PCI-MSI         status=00000010 in-flight=0
domain-list=0:289(----),
(XEN)    IRQ:  64 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:99
type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  65 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:b1
type=PCI-MSI         status=00000010 in-flight=0
domain-list=0:287(----),
(XEN) IO-APIC interrupt information:

''q''
(XEN)     Interrupts { 32, 41-42, 47 }
>
>>
>> > >
>> > > kaan-18-dpm:~# cat /proc/interrupts | grep TACH
>> > > 124:     760415          0          0          0          0  
       0
>> > >          0          0          0          0          0      
   0
>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
>> > > 125:     762234          0          0          0          0  
       0
>> > >          0          0          0          0          0      
   0
>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
>> > > 126:     764180          0          0          0          0  
       0
>> > >          0          0          0          0          0      
   0
>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
>> > > 127:     764164          0          0          0          0  
       0
>> > >          0          0          0          0          0      
   0
>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
>> >
>> > Can you provide the full dmesg output?
>>
>> Attached.
>>
>> Some possibly related messages on dom0 console:
>>
>> [ 1882.269778] pciback 0000:07:00.0: enabling device (0000 -> 0003)
>> [ 1882.269800] xen: registering gsi 32 triggering 0 polarity 1
>> [ 1882.269827] xen_allocate_pirq: returning irq 32 for gsi 32
>> [ 1882.269834] xen: --> irq=32
>> [ 1882.269841] Already setup the GSI :32
>> [ 1882.269847] pciback 0000:07:00.0: PCI INT A -> GSI 32 (level,
low) -> IRQ 32
>> [ 1882.269866] pciback 0000:07:00.0: setting latency timer to 64
>> [ 1882.270463] pciback 0000:07:00.0: Driver tried to write to a
>> read-only configuration space field at offset 0x62, size 2. This may
>> be harmless, but if you have problems with your device:
>
> Uhhh, for that I think you need to do ''lspci -vvv -xxx -s
07:00.00''
> to find out what is at the configuration space. You could enable
> it using the permissive attribute.
>
>> [ 1882.270465] 1) see permissive attribute in sysfs
>> [ 1882.270467] 2) report problems to the xen-devel mailing list along
>> with details of your device obtained from lspci.
>> [ 1882.270615]   alloc irq_desc for 478 on node 0
>> [ 1882.270625]   alloc kstat_irqs on node 0
>
> So for 478: what do you see? xen-pciback I presume?
>> [ 1882.348411] pciback 0000:07:00.1: enabling device (0000 -> 0003)
>> [ 1882.348433] xen: registering gsi 42 triggering 0 polarity 1
>> [ 1882.348440] xen_allocate_pirq: returning irq 42 for gsi 42
>> [ 1882.348445] xen: --> irq=42
>> [ 1882.348472] Already setup the GSI :42
>> [ 1882.348479] pciback 0000:07:00.1: PCI INT B -> GSI 42 (level,
low) -> IRQ 42
>> [ 1882.348497] pciback 0000:07:00.1: setting latency timer to 64
>> [ 1882.349063] pciback 0000:07:00.1: Driver tried to write to a
>> read-only configuration space field at offset 0x62, size 2. This may
>> be harmless, but if you have problems with your device:
>> [ 1882.349066] 1) see permissive attribute in sysfs
>> [ 1882.349067] 2) report problems to the xen-devel mailing list along
>> with details of your device obtained from lspci.
>> [ 1882.349205]   alloc irq_desc for 477 on node 0
>> [ 1882.349215]   alloc kstat_irqs on node 0
>> [ 1882.402893] pciback 0000:07:00.2: enabling device (0000 -> 0003)
>> [ 1882.402908] xen: registering gsi 47 triggering 0 polarity 1
>> [ 1882.402913] xen_allocate_pirq: returning irq 47 for gsi 47
>> [ 1882.402916] xen: --> irq=47
>> [ 1882.402921] Already setup the GSI :47
>> [ 1882.402925] pciback 0000:07:00.2: PCI INT C -> GSI 47 (level,
low) -> IRQ 47
>> [ 1882.402938] pciback 0000:07:00.2: setting latency timer to 64
>> [ 1882.403280] pciback 0000:07:00.2: Driver tried to write to a
>> read-only configuration space field at offset 0x62, size 2. This may
>> be harmless, but if you have problems with your device:
>> [ 1882.403282] 1) see permissive attribute in sysfs
>> [ 1882.403282] 2) report problems to the xen-devel mailing list along
>> with details of your device obtained from lspci.
>> [ 1882.403380]   alloc irq_desc for 476 on node 0
>> [ 1882.403386]   alloc kstat_irqs on node 0
>> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
>> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
>> fault addr e6f80000, iommu reg = ffff82c3fff57000
>> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
>> (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn =
e6f80
>> (XEN)     root_entry = ffff83019ff70000
>> (XEN)     root_entry[7] = 19cf52001
>> (XEN)     context = ffff83019cf52000
>> (XEN)     context[0] = 102_706dc005
>> (XEN)     l4 = ffff8300706dc000
>> (XEN)     l4_index = 0
>> (XEN)     l4[0] = 706db003
>> (XEN)     l3 = ffff8300706db000
>> (XEN)     l3_index = 3
>> (XEN)     l3[3] = 702b6003
>> (XEN)     l2 = ffff8300702b6000
>> (XEN)     l2_index = 137
>> (XEN)     l2[137] = 0
>> (XEN)     l2[137] not present
>> (XEN) traps.c:466:d0 Unhandled nmi fault/trap [#2] on VCPU 0 [ec=0000]
>
> That is not good. What changed from your earlier emails that this was
triggered?
Nothing> Or was it triggered all along?
Yes, I just included it for completeness
> What happens if you run the system without the iommu enabled?
Haven''t tried yet. Will check that next.

-Bruce

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Jiang, Yunhong

2010-Sep-28 01:15 UTC

head link

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Is the 07:0.0 your tachyon device? The VT-d fault is suspcious.
Also is it possible to share the xen output?

Thanks
--jyh
>-----Original Message-----
>From: xen-devel-bounces@lists.xensource.com
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Bruce Edge
>Sent: Tuesday, September 28, 2010 7:54 AM
>To: Konrad Rzeszutek Wilk
>Cc: xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on
Nehalem
>
>On Mon, Sep 27, 2010 at 12:54 PM, Konrad Rzeszutek Wilk
><konrad.wilk@oracle.com> wrote:
>> On Mon, Sep 27, 2010 at 12:16:50PM -0700, Bruce Edge wrote:
>>> On Mon, Sep 27, 2010 at 10:24 AM, Konrad Rzeszutek Wilk
>>> <konrad.wilk@oracle.com> wrote:
>>> >
>>> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge wrote:
>>> > > One of our developers who is working on a tachyon driver
is
>>> > > complaining that the pvops domU kernel is not working for
these MSI
>>> > > interrupts.
>>> > > This is using the current head of xen/2.6.32.x on both a
single
>>> > > Nahelam 920 and a dual E5540. This behavior is consistent
with Xen
>>> > > 4.0.1, 4.0.2.rc1-pre and 4.1.
>>> > >
>>> > > Here are his comments:
>>> > >
>>> > > - the driver has no problem to enable msi interrupt and
request the
>>> > > interrupt through kernel functions pci_enable_msi &
request_irq
>>> >
>>> > What shows up in the Xen console when you send the
''q'' key? Does it
>>> > show that the vector is assigned to the appropiate guest?
>>>
>>> The Xen console q key shows that the domU is assigned:
>>>
>>> (XEN)     Interrupts { 32, 41-42, 47 }
>>
>> Aha!
>>
>>>
>>> but the domU thinks it has:
>>>
>>> 124/125/126/127
>>>
>>> Is there some mapping that''s taking place, or is this
plain wrong?
>>
>> That looks wrong. The IRQ numbers (even though they are MSI vectors)
are
>> setup as IRQ numbers in the DomU guest. You should have seen
>>
>> 32:
>> 41:
>> 42:
>> 47:
>> in you /proc/interrupts on your DomU guest.
>>
>> I wonder what broke  - can you use
>git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
>> devel/xen-pcifront-0.5 (or pv/pcifront-2.6.32)?
>
>Please forgive the git ignorance.
>
>Is this the right syntax?
>
>git clone
git://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.32
>linux-2.6.32-pv-pcifront
>
>Initialized empty Git repository in
>/import/kaan/bedge/src/xen/kernel/pv-ops/linux-2.6.32-pv-pcifront/.git/
>fatal: The remote end hung up unexpectedly
>
>Or:
>
> git clone  git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
>
>Initialized empty Git repository in
>/import/kaan/bedge/src/xen/kernel/pv-ops/xen/.git/
>remote: error: Could not read 59eab2f8f04147c5aadc99f2034ca7e5b81e890f
>remote: fatal: Failed to traverse parents of commit
>979e121cb348add17ed8171bf447b27a3a9d1be3
>remote: aborting due to possible repository corruption on the remote side.
>fatal: early EOF
>fatal: index-pack failed
>
>>
>> It has the latest pcifront driver but without the PVonHVM enhancments
>> so we can try to eliminate the PvONHVM logic out of the picture.
>>
>>>
>>> >
>>> > > - the interrupt does happen. But the interrupt service
routine of
>>> > > tachyon driver doesn''t detect any interrupt
status related to this
>>> > > interrupt, which inhibits the tachyon chip from coming
on-line. And
>>> > > there are high count of tachyon interrupt in
/proc/interrupts
>>> >
>>> > Is it checking the PCI_STATUS_INTERRUPT or the appropiate
register
>>> > in the MMIO BAR?
>>> >
>>>
>>> The driver would check the appropriate register (tachyon registers)
in
>>> the MMIO to determine the source of interrupts.
>>
>> OK, so that isn''t it. Is there anything at these vectors:
>> 7c, 7d, 7e, and 7f? When you use xen debug-keys ''i''
or ''q'' it should give you
>> an inkling what device this is set for.
>
>When I run a distro kernel in hvm mode, I get the expected irq mappings:
>
>''i'' - Note 66 - 69
>(XEN)    IRQ:  66 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:3a
>type=PCI-MSI         status=00000010 in-flight=0
>domain-list=10:127(----),
>(XEN)    IRQ:  67 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:42
>type=PCI-MSI         status=00000010 in-flight=0
>domain-list=10:126(----),
>(XEN)    IRQ:  68 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:4a
>type=PCI-MSI         status=00000010 in-flight=0
>domain-list=10:125(----),
>(XEN)    IRQ:  69 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:52
>type=PCI-MSI         status=00000010 in-flight=0
>domain-list=10:124(----)
>
>
>''q''
>(XEN)     Interrupts { 32, 41-42, 47, 124-127 }
>
>
>The same data with pv-ops kernel shows:
>
>''i''
>IRQ numbers stop at 65, no 66 - 69 present:
>
>(XEN)    IRQ:  63 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:91
>type=PCI-MSI         status=00000010 in-flight=0
>domain-list=0:289(----),
>(XEN)    IRQ:  64 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:99
>type=PCI-MSI         status=00000002 mapped, unbound
>(XEN)    IRQ:  65 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:b1
>type=PCI-MSI         status=00000010 in-flight=0
>domain-list=0:287(----),
>(XEN) IO-APIC interrupt information:
>
>''q''
>(XEN)     Interrupts { 32, 41-42, 47 }
>
>>
>>>
>>> > >
>>> > > kaan-18-dpm:~# cat /proc/interrupts | grep TACH
>>> > >
>124:     760415          0          0          0          0
>    0
>>> > >          0          0          0          0          0
>      0
>>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
>>> > >
>125:     762234          0          0          0          0
>    0
>>> > >          0          0          0          0          0
>      0
>>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
>>> > >
>126:     764180          0          0          0          0
>    0
>>> > >          0          0          0          0          0
>      0
>>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
>>> > >
>127:     764164          0          0          0          0
>    0
>>> > >          0          0          0          0          0
>      0
>>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
>>> >
>>> > Can you provide the full dmesg output?
>>>
>>> Attached.
>>>
>>> Some possibly related messages on dom0 console:
>>>
>>> [ 1882.269778] pciback 0000:07:00.0: enabling device (0000 ->
0003)
>>> [ 1882.269800] xen: registering gsi 32 triggering 0 polarity 1
>>> [ 1882.269827] xen_allocate_pirq: returning irq 32 for gsi 32
>>> [ 1882.269834] xen: --> irq=32
>>> [ 1882.269841] Already setup the GSI :32
>>> [ 1882.269847] pciback 0000:07:00.0: PCI INT A -> GSI 32 (level,
low) -> IRQ 32
>>> [ 1882.269866] pciback 0000:07:00.0: setting latency timer to 64
>>> [ 1882.270463] pciback 0000:07:00.0: Driver tried to write to a
>>> read-only configuration space field at offset 0x62, size 2. This
may
>>> be harmless, but if you have problems with your device:
>>
>> Uhhh, for that I think you need to do ''lspci -vvv -xxx -s
07:00.00''
>> to find out what is at the configuration space. You could enable
>> it using the permissive attribute.
>>
>>> [ 1882.270465] 1) see permissive attribute in sysfs
>>> [ 1882.270467] 2) report problems to the xen-devel mailing list
along
>>> with details of your device obtained from lspci.
>>> [ 1882.270615]   alloc irq_desc for 478 on node 0
>>> [ 1882.270625]   alloc kstat_irqs on node 0
>>
>> So for 478: what do you see? xen-pciback I presume?
>>> [ 1882.348411] pciback 0000:07:00.1: enabling device (0000 ->
0003)
>>> [ 1882.348433] xen: registering gsi 42 triggering 0 polarity 1
>>> [ 1882.348440] xen_allocate_pirq: returning irq 42 for gsi 42
>>> [ 1882.348445] xen: --> irq=42
>>> [ 1882.348472] Already setup the GSI :42
>>> [ 1882.348479] pciback 0000:07:00.1: PCI INT B -> GSI 42 (level,
low) -> IRQ 42
>>> [ 1882.348497] pciback 0000:07:00.1: setting latency timer to 64
>>> [ 1882.349063] pciback 0000:07:00.1: Driver tried to write to a
>>> read-only configuration space field at offset 0x62, size 2. This
may
>>> be harmless, but if you have problems with your device:
>>> [ 1882.349066] 1) see permissive attribute in sysfs
>>> [ 1882.349067] 2) report problems to the xen-devel mailing list
along
>>> with details of your device obtained from lspci.
>>> [ 1882.349205]   alloc irq_desc for 477 on node 0
>>> [ 1882.349215]   alloc kstat_irqs on node 0
>>> [ 1882.402893] pciback 0000:07:00.2: enabling device (0000 ->
0003)
>>> [ 1882.402908] xen: registering gsi 47 triggering 0 polarity 1
>>> [ 1882.402913] xen_allocate_pirq: returning irq 47 for gsi 47
>>> [ 1882.402916] xen: --> irq=47
>>> [ 1882.402921] Already setup the GSI :47
>>> [ 1882.402925] pciback 0000:07:00.2: PCI INT C -> GSI 47 (level,
low) -> IRQ 47
>>> [ 1882.402938] pciback 0000:07:00.2: setting latency timer to 64
>>> [ 1882.403280] pciback 0000:07:00.2: Driver tried to write to a
>>> read-only configuration space field at offset 0x62, size 2. This
may
>>> be harmless, but if you have problems with your device:
>>> [ 1882.403282] 1) see permissive attribute in sysfs
>>> [ 1882.403282] 2) report problems to the xen-devel mailing list
along
>>> with details of your device obtained from lspci.
>>> [ 1882.403380]   alloc irq_desc for 476 on node 0
>>> [ 1882.403386]   alloc kstat_irqs on node 0
>>> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
>>> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
>>> fault addr e6f80000, iommu reg = ffff82c3fff57000
>>> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
>>> (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn
= e6f80
>>> (XEN)     root_entry = ffff83019ff70000
>>> (XEN)     root_entry[7] = 19cf52001
>>> (XEN)     context = ffff83019cf52000
>>> (XEN)     context[0] = 102_706dc005
>>> (XEN)     l4 = ffff8300706dc000
>>> (XEN)     l4_index = 0
>>> (XEN)     l4[0] = 706db003
>>> (XEN)     l3 = ffff8300706db000
>>> (XEN)     l3_index = 3
>>> (XEN)     l3[3] = 702b6003
>>> (XEN)     l2 = ffff8300702b6000
>>> (XEN)     l2_index = 137
>>> (XEN)     l2[137] = 0
>>> (XEN)     l2[137] not present
>>> (XEN) traps.c:466:d0 Unhandled nmi fault/trap [#2] on VCPU 0
[ec=0000]
>>
>> That is not good. What changed from your earlier emails that this was
triggered?
>
>Nothing
>> Or was it triggered all along?
>
>Yes, I just included it for completeness
>
>> What happens if you run the system without the iommu enabled?
>
>Haven''t tried yet. Will check that next.
>
>-Bruce
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Bruce Edge

2010-Sep-28 03:16 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Mon, Sep 27, 2010 at 6:15 PM, Jiang, Yunhong
<yunhong.jiang@intel.com>wrote:
> Is the 07:0.0 your tachyon device? The VT-d fault is suspcious.
>
Yes, there is 1 quad port card is this sytem:

07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
07:00.1 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
07:00.2 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
07:00.3 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)


> Also is it possible to share the xen output?
>
I attached the dom0 boot output. Let me know if you wanted something else.

Also, here''s the dom0 console output upon starting the VM: This lockdep
error started with the release of 2.6.32.21. Note that I''m running the
same
 kernel for the domU and dom0.

[ 1817.684097] ------------[ cut here ]------------
[ 1817.684113] WARNING: at kernel/lockdep.c:2323
trace_hardirqs_on_caller+0x12f/0x190()
[ 1817.684119] Hardware name: ProLiant DL380 G6
[ 1817.684122] Modules linked in: xt_physdev ipv6 osa_mfgdom0 xenfs
xen_gntdev fbcon tileblit font bitblit softcursor xen_evtchn xen_pciback
radeon ttm drm_kms_helper tun drm i2c_algo_bit ipmi_si i2c_core
ipmi_msghandler joydev serio_raw hpwdt hpilo bridge stp llc usbhid hid cciss
usb_storage
[ 1817.684190] Pid: 11, comm: xenwatch Not tainted 2.6.32.21-xenoprof-1 #1
[ 1817.684195] Call Trace:
[ 1817.684197]  <IRQ>  [<ffffffff810aa18f>] ?
trace_hardirqs_on_caller+0x12f/0x190
[ 1817.684209]  [<ffffffff8106bed0>] warn_slowpath_common+0x80/0xd0
[ 1817.684217]  [<ffffffff815f2b80>] ? _spin_unlock_irq+0x30/0x40
[ 1817.684223]  [<ffffffff8106bf34>] warn_slowpath_null+0x14/0x20
[ 1817.684229]  [<ffffffff810aa18f>] trace_hardirqs_on_caller+0x12f/0x190
[ 1817.684234]  [<ffffffff810aa1fd>] trace_hardirqs_on+0xd/0x10
[ 1817.684240]  [<ffffffff815f2b80>] _spin_unlock_irq+0x30/0x40
[ 1817.684266]  [<ffffffff813c4fc5>]
add_to_net_schedule_list_tail+0x85/0xd0
[ 1817.684271]  [<ffffffff813c6216>] netif_be_int+0x36/0x160
[ 1817.684278]  [<ffffffff810e10d0>] handle_IRQ_event+0x70/0x180
[ 1817.684284]  [<ffffffff810e36e9>] handle_edge_irq+0xc9/0x170
[ 1817.684291]  [<ffffffff813b8d7f>] __xen_evtchn_do_upcall+0x1bf/0x1f0
[ 1817.684297]  [<ffffffff813b92fd>] xen_evtchn_do_upcall+0x3d/0x60
[ 1817.684304]  [<ffffffff8101647e>] xen_do_hypervisor_callback+0x1e/0x30
[ 1817.684308]  <EOI>  [<ffffffff8100940a>] ?
hypercall_page+0x40a/0x1010
[ 1817.684319]  [<ffffffff8100940a>] ? hypercall_page+0x40a/0x1010
[ 1817.684325]  [<ffffffff813bce54>] ? xb_write+0x1e4/0x290
[ 1817.684330]  [<ffffffff813bd8ca>] ? xs_talkv+0x6a/0x1f0
[ 1817.684336]  [<ffffffff813bd8d8>] ? xs_talkv+0x78/0x1f0
[ 1817.684341]  [<ffffffff813bdbcd>] ? xs_single+0x4d/0x60
[ 1817.684346]  [<ffffffff813be502>] ? xenbus_read+0x52/0x80
[ 1817.684352]  [<ffffffff813c87fc>] ? frontend_changed+0x48c/0x770
[ 1817.684358]  [<ffffffff813bf76d>] ? xenbus_otherend_changed+0xdd/0x1b0
[ 1817.684365]  [<ffffffff8101122f>] ? xen_restore_fl_direct_end+0x0/0x1
[ 1817.684371]  [<ffffffff810ac830>] ? lock_release+0xb0/0x230
[ 1817.684376]  [<ffffffff813bfae0>] ? frontend_changed+0x10/0x20
[ 1817.684382]  [<ffffffff813bd4f5>] ? xenwatch_thread+0x55/0x160
[ 1817.684389]  [<ffffffff81093400>] ? autoremove_wake_function+0x0/0x40
[ 1817.684394]  [<ffffffff813bd4a0>] ? xenwatch_thread+0x0/0x160
[ 1817.684400]  [<ffffffff81093086>] ? kthread+0x96/0xb0
[ 1817.684405]  [<ffffffff8101632a>] ? child_rip+0xa/0x20
[ 1817.684410]  [<ffffffff81015c90>] ? restore_args+0x0/0x30
[ 1817.684415]  [<ffffffff81016320>] ? child_rip+0x0/0x20

-Bruce


>
> Thanks
> --jyh
>
> >-----Original Message-----
> >From: xen-devel-bounces@lists.xensource.com
> >[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Bruce Edge
> >Sent: Tuesday, September 28, 2010 7:54 AM
> >To: Konrad Rzeszutek Wilk
> >Cc: xen-devel@lists.xensource.com
> >Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on
> Nehalem
> >
> >On Mon, Sep 27, 2010 at 12:54 PM, Konrad Rzeszutek Wilk
> ><konrad.wilk@oracle.com> wrote:
> >> On Mon, Sep 27, 2010 at 12:16:50PM -0700, Bruce Edge wrote:
> >>> On Mon, Sep 27, 2010 at 10:24 AM, Konrad Rzeszutek Wilk
> >>> <konrad.wilk@oracle.com> wrote:
> >>> >
> >>> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge
wrote:
> >>> > > One of our developers who is working on a tachyon
driver is
> >>> > > complaining that the pvops domU kernel is not
working for these MSI
> >>> > > interrupts.
> >>> > > This is using the current head of xen/2.6.32.x on
both a single
> >>> > > Nahelam 920 and a dual E5540. This behavior is
consistent with Xen
> >>> > > 4.0.1, 4.0.2.rc1-pre and 4.1.
> >>> > >
> >>> > > Here are his comments:
> >>> > >
> >>> > > - the driver has no problem to enable msi interrupt
and request the
> >>> > > interrupt through kernel functions pci_enable_msi
& request_irq
> >>> >
> >>> > What shows up in the Xen console when you send the
''q'' key? Does it
> >>> > show that the vector is assigned to the appropiate guest?
> >>>
> >>> The Xen console q key shows that the domU is assigned:
> >>>
> >>> (XEN)     Interrupts { 32, 41-42, 47 }
> >>
> >> Aha!
> >>
> >>>
> >>> but the domU thinks it has:
> >>>
> >>> 124/125/126/127
> >>>
> >>> Is there some mapping that''s taking place, or is this
plain wrong?
> >>
> >> That looks wrong. The IRQ numbers (even though they are MSI
vectors) are
> >> setup as IRQ numbers in the DomU guest. You should have seen
> >>
> >> 32:
> >> 41:
> >> 42:
> >> 47:
> >> in you /proc/interrupts on your DomU guest.
> >>
> >> I wonder what broke  - can you use
> >git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
> >> devel/xen-pcifront-0.5 (or pv/pcifront-2.6.32)?
> >
> >Please forgive the git ignorance.
> >
> >Is this the right syntax?
> >
> >git clone git://
> git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.32
> >linux-2.6.32-pv-pcifront
> >
> >Initialized empty Git repository in
> >/import/kaan/bedge/src/xen/kernel/pv-ops/linux-2.6.32-pv-pcifront/.git/
> >fatal: The remote end hung up unexpectedly
> >
> >Or:
> >
> > git clone 
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
> >
> >Initialized empty Git repository in
> >/import/kaan/bedge/src/xen/kernel/pv-ops/xen/.git/
> >remote: error: Could not read 59eab2f8f04147c5aadc99f2034ca7e5b81e890f
> >remote: fatal: Failed to traverse parents of commit
> >979e121cb348add17ed8171bf447b27a3a9d1be3
> >remote: aborting due to possible repository corruption on the remote
side.
> >fatal: early EOF
> >fatal: index-pack failed
> >
> >>
> >> It has the latest pcifront driver but without the PVonHVM
enhancments
> >> so we can try to eliminate the PvONHVM logic out of the picture.
> >>
> >>>
> >>> >
> >>> > > - the interrupt does happen. But the interrupt
service routine of
> >>> > > tachyon driver doesn''t detect any interrupt
status related to this
> >>> > > interrupt, which inhibits the tachyon chip from
coming on-line. And
> >>> > > there are high count of tachyon interrupt in
/proc/interrupts
> >>> >
> >>> > Is it checking the PCI_STATUS_INTERRUPT or the appropiate
register
> >>> > in the MMIO BAR?
> >>> >
> >>>
> >>> The driver would check the appropriate register (tachyon
registers) in
> >>> the MMIO to determine the source of interrupts.
> >>
> >> OK, so that isn''t it. Is there anything at these vectors:
> >> 7c, 7d, 7e, and 7f? When you use xen debug-keys
''i'' or ''q'' it should
> give you
> >> an inkling what device this is set for.
> >
> >When I run a distro kernel in hvm mode, I get the expected irq
mappings:
> >
> >''i'' - Note 66 - 69
> >(XEN)    IRQ:  66 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:3a
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=10:127(----),
> >(XEN)    IRQ:  67 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:42
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=10:126(----),
> >(XEN)    IRQ:  68 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:4a
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=10:125(----),
> >(XEN)    IRQ:  69 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:52
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=10:124(----)
> >
> >
> >''q''
> >(XEN)     Interrupts { 32, 41-42, 47, 124-127 }
> >
> >
> >The same data with pv-ops kernel shows:
> >
> >''i''
> >IRQ numbers stop at 65, no 66 - 69 present:
> >
> >(XEN)    IRQ:  63 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:91
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=0:289(----),
> >(XEN)    IRQ:  64 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:99
> >type=PCI-MSI         status=00000002 mapped, unbound
> >(XEN)    IRQ:  65 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:b1
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=0:287(----),
> >(XEN) IO-APIC interrupt information:
> >
> >''q''
> >(XEN)     Interrupts { 32, 41-42, 47 }
> >
> >>
> >>>
> >>> > >
> >>> > > kaan-18-dpm:~# cat /proc/interrupts | grep TACH
> >>> > >
> >124:     760415          0          0          0          0
> >    0
> >>> > >          0          0          0          0         
0
> >      0
> >>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> >>> > >
> >125:     762234          0          0          0          0
> >    0
> >>> > >          0          0          0          0         
0
> >      0
> >>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> >>> > >
> >126:     764180          0          0          0          0
> >    0
> >>> > >          0          0          0          0         
0
> >      0
> >>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> >>> > >
> >127:     764164          0          0          0          0
> >    0
> >>> > >          0          0          0          0         
0
> >      0
> >>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> >>> >
> >>> > Can you provide the full dmesg output?
> >>>
> >>> Attached.
> >>>
> >>> Some possibly related messages on dom0 console:
> >>>
> >>> [ 1882.269778] pciback 0000:07:00.0: enabling device (0000
-> 0003)
> >>> [ 1882.269800] xen: registering gsi 32 triggering 0 polarity 1
> >>> [ 1882.269827] xen_allocate_pirq: returning irq 32 for gsi 32
> >>> [ 1882.269834] xen: --> irq=32
> >>> [ 1882.269841] Already setup the GSI :32
> >>> [ 1882.269847] pciback 0000:07:00.0: PCI INT A -> GSI 32
(level, low)
> -> IRQ 32
> >>> [ 1882.269866] pciback 0000:07:00.0: setting latency timer to
64
> >>> [ 1882.270463] pciback 0000:07:00.0: Driver tried to write to
a
> >>> read-only configuration space field at offset 0x62, size 2.
This may
> >>> be harmless, but if you have problems with your device:
> >>
> >> Uhhh, for that I think you need to do ''lspci -vvv -xxx -s
07:00.00''
> >> to find out what is at the configuration space. You could enable
> >> it using the permissive attribute.
> >>
> >>> [ 1882.270465] 1) see permissive attribute in sysfs
> >>> [ 1882.270467] 2) report problems to the xen-devel mailing
list along
> >>> with details of your device obtained from lspci.
> >>> [ 1882.270615]   alloc irq_desc for 478 on node 0
> >>> [ 1882.270625]   alloc kstat_irqs on node 0
> >>
> >> So for 478: what do you see? xen-pciback I presume?
> >>> [ 1882.348411] pciback 0000:07:00.1: enabling device (0000
-> 0003)
> >>> [ 1882.348433] xen: registering gsi 42 triggering 0 polarity 1
> >>> [ 1882.348440] xen_allocate_pirq: returning irq 42 for gsi 42
> >>> [ 1882.348445] xen: --> irq=42
> >>> [ 1882.348472] Already setup the GSI :42
> >>> [ 1882.348479] pciback 0000:07:00.1: PCI INT B -> GSI 42
(level, low)
> -> IRQ 42
> >>> [ 1882.348497] pciback 0000:07:00.1: setting latency timer to
64
> >>> [ 1882.349063] pciback 0000:07:00.1: Driver tried to write to
a
> >>> read-only configuration space field at offset 0x62, size 2.
This may
> >>> be harmless, but if you have problems with your device:
> >>> [ 1882.349066] 1) see permissive attribute in sysfs
> >>> [ 1882.349067] 2) report problems to the xen-devel mailing
list along
> >>> with details of your device obtained from lspci.
> >>> [ 1882.349205]   alloc irq_desc for 477 on node 0
> >>> [ 1882.349215]   alloc kstat_irqs on node 0
> >>> [ 1882.402893] pciback 0000:07:00.2: enabling device (0000
-> 0003)
> >>> [ 1882.402908] xen: registering gsi 47 triggering 0 polarity 1
> >>> [ 1882.402913] xen_allocate_pirq: returning irq 47 for gsi 47
> >>> [ 1882.402916] xen: --> irq=47
> >>> [ 1882.402921] Already setup the GSI :47
> >>> [ 1882.402925] pciback 0000:07:00.2: PCI INT C -> GSI 47
(level, low)
> -> IRQ 47
> >>> [ 1882.402938] pciback 0000:07:00.2: setting latency timer to
64
> >>> [ 1882.403280] pciback 0000:07:00.2: Driver tried to write to
a
> >>> read-only configuration space field at offset 0x62, size 2.
This may
> >>> be harmless, but if you have problems with your device:
> >>> [ 1882.403282] 1) see permissive attribute in sysfs
> >>> [ 1882.403282] 2) report problems to the xen-devel mailing
list along
> >>> with details of your device obtained from lspci.
> >>> [ 1882.403380]   alloc irq_desc for 476 on node 0
> >>> [ 1882.403386]   alloc kstat_irqs on node 0
> >>> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending
Fault
> >>> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device
[07:00.0]
> >>> fault addr e6f80000, iommu reg = ffff82c3fff57000
> >>> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
> >>> (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0
gmfn > e6f80
> >>> (XEN)     root_entry = ffff83019ff70000
> >>> (XEN)     root_entry[7] = 19cf52001
> >>> (XEN)     context = ffff83019cf52000
> >>> (XEN)     context[0] = 102_706dc005
> >>> (XEN)     l4 = ffff8300706dc000
> >>> (XEN)     l4_index = 0
> >>> (XEN)     l4[0] = 706db003
> >>> (XEN)     l3 = ffff8300706db000
> >>> (XEN)     l3_index = 3
> >>> (XEN)     l3[3] = 702b6003
> >>> (XEN)     l2 = ffff8300702b6000
> >>> (XEN)     l2_index = 137
> >>> (XEN)     l2[137] = 0
> >>> (XEN)     l2[137] not present
> >>> (XEN) traps.c:466:d0 Unhandled nmi fault/trap [#2] on VCPU 0
[ec=0000]
> >>
> >> That is not good. What changed from your earlier emails that this
was
> triggered?
> >
> >Nothing
> >> Or was it triggered all along?
> >
> >Yes, I just included it for completeness
> >
> >> What happens if you run the system without the iommu enabled?
> >
> >Haven''t tried yet. Will check that next.
> >
> >-Bruce
> >
> >_______________________________________________
> >Xen-devel mailing list
> >Xen-devel@lists.xensource.com
> >http://lists.xensource.com/xen-devel
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Jiang, Yunhong

2010-Sep-28 03:26 UTC

head link

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

"xm dmesg" should gives xen''s boot log, and sometimes it
contain some helpful information, I think, especially loglvl and guest_loglvl is
set to all.

Thanks
--jyh

From: Bruce Edge [mailto:bruce.edge@gmail.com]
Sent: Tuesday, September 28, 2010 11:16 AM
To: Jiang, Yunhong
Cc: Konrad Rzeszutek Wilk; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Mon, Sep 27, 2010 at 6:15 PM, Jiang, Yunhong
<yunhong.jiang@intel.com<mailto:yunhong.jiang@intel.com>> wrote:
Is the 07:0.0 your tachyon device? The VT-d fault is suspcious.

Yes, there is 1 quad port card is this sytem:

07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
07:00.1 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
07:00.2 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
07:00.3 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)


Also is it possible to share the xen output?

I attached the dom0 boot output. Let me know if you wanted something else.

Also, here''s the dom0 console output upon starting the VM: This lockdep
error started with the release of 2.6.32.21. Note that I''m running the
same  kernel for the domU and dom0.

[ 1817.684097] ------------[ cut here ]------------
[ 1817.684113] WARNING: at kernel/lockdep.c:2323
trace_hardirqs_on_caller+0x12f/0x190()
[ 1817.684119] Hardware name: ProLiant DL380 G6
[ 1817.684122] Modules linked in: xt_physdev ipv6 osa_mfgdom0 xenfs xen_gntdev
fbcon tileblit font bitblit softcursor xen_evtchn xen_pciback radeon ttm
drm_kms_helper tun drm i2c_algo_bit ipmi_si i2c_core ipmi_msghandler joydev
serio_raw hpwdt hpilo bridge stp llc usbhid hid cciss usb_storage
[ 1817.684190] Pid: 11, comm: xenwatch Not tainted 2.6.32.21-xenoprof-1 #1
[ 1817.684195] Call Trace:
[ 1817.684197]  <IRQ>  [<ffffffff810aa18f>] ?
trace_hardirqs_on_caller+0x12f/0x190
[ 1817.684209]  [<ffffffff8106bed0>] warn_slowpath_common+0x80/0xd0
[ 1817.684217]  [<ffffffff815f2b80>] ? _spin_unlock_irq+0x30/0x40
[ 1817.684223]  [<ffffffff8106bf34>] warn_slowpath_null+0x14/0x20
[ 1817.684229]  [<ffffffff810aa18f>] trace_hardirqs_on_caller+0x12f/0x190
[ 1817.684234]  [<ffffffff810aa1fd>] trace_hardirqs_on+0xd/0x10
[ 1817.684240]  [<ffffffff815f2b80>] _spin_unlock_irq+0x30/0x40
[ 1817.684266]  [<ffffffff813c4fc5>]
add_to_net_schedule_list_tail+0x85/0xd0
[ 1817.684271]  [<ffffffff813c6216>] netif_be_int+0x36/0x160
[ 1817.684278]  [<ffffffff810e10d0>] handle_IRQ_event+0x70/0x180
[ 1817.684284]  [<ffffffff810e36e9>] handle_edge_irq+0xc9/0x170
[ 1817.684291]  [<ffffffff813b8d7f>] __xen_evtchn_do_upcall+0x1bf/0x1f0
[ 1817.684297]  [<ffffffff813b92fd>] xen_evtchn_do_upcall+0x3d/0x60
[ 1817.684304]  [<ffffffff8101647e>] xen_do_hypervisor_callback+0x1e/0x30
[ 1817.684308]  <EOI>  [<ffffffff8100940a>] ?
hypercall_page+0x40a/0x1010
[ 1817.684319]  [<ffffffff8100940a>] ? hypercall_page+0x40a/0x1010
[ 1817.684325]  [<ffffffff813bce54>] ? xb_write+0x1e4/0x290
[ 1817.684330]  [<ffffffff813bd8ca>] ? xs_talkv+0x6a/0x1f0
[ 1817.684336]  [<ffffffff813bd8d8>] ? xs_talkv+0x78/0x1f0
[ 1817.684341]  [<ffffffff813bdbcd>] ? xs_single+0x4d/0x60
[ 1817.684346]  [<ffffffff813be502>] ? xenbus_read+0x52/0x80
[ 1817.684352]  [<ffffffff813c87fc>] ? frontend_changed+0x48c/0x770
[ 1817.684358]  [<ffffffff813bf76d>] ? xenbus_otherend_changed+0xdd/0x1b0
[ 1817.684365]  [<ffffffff8101122f>] ? xen_restore_fl_direct_end+0x0/0x1
[ 1817.684371]  [<ffffffff810ac830>] ? lock_release+0xb0/0x230
[ 1817.684376]  [<ffffffff813bfae0>] ? frontend_changed+0x10/0x20
[ 1817.684382]  [<ffffffff813bd4f5>] ? xenwatch_thread+0x55/0x160
[ 1817.684389]  [<ffffffff81093400>] ? autoremove_wake_function+0x0/0x40
[ 1817.684394]  [<ffffffff813bd4a0>] ? xenwatch_thread+0x0/0x160
[ 1817.684400]  [<ffffffff81093086>] ? kthread+0x96/0xb0
[ 1817.684405]  [<ffffffff8101632a>] ? child_rip+0xa/0x20
[ 1817.684410]  [<ffffffff81015c90>] ? restore_args+0x0/0x30
[ 1817.684415]  [<ffffffff81016320>] ? child_rip+0x0/0x20

-Bruce



Thanks
--jyh
>-----Original Message-----
>From:
xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounces@lists.xensource.com>
>[mailto:xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounces@lists.xensource.com>]
On Behalf Of Bruce Edge
>Sent: Tuesday, September 28, 2010 7:54 AM
>To: Konrad Rzeszutek Wilk
>Cc:
xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>
>Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on
Nehalem
>
>On Mon, Sep 27, 2010 at 12:54 PM, Konrad Rzeszutek Wilk
><konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>> wrote:
>> On Mon, Sep 27, 2010 at 12:16:50PM -0700, Bruce Edge wrote:
>>> On Mon, Sep 27, 2010 at 10:24 AM, Konrad Rzeszutek Wilk
>>> <konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>>
wrote:
>>> >
>>> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge wrote:
>>> > > One of our developers who is working on a tachyon driver
is
>>> > > complaining that the pvops domU kernel is not working for
these MSI
>>> > > interrupts.
>>> > > This is using the current head of xen/2.6.32.x on both a
single
>>> > > Nahelam 920 and a dual E5540. This behavior is consistent
with Xen
>>> > > 4.0.1, 4.0.2.rc1-pre and 4.1.
>>> > >
>>> > > Here are his comments:
>>> > >
>>> > > - the driver has no problem to enable msi interrupt and
request the
>>> > > interrupt through kernel functions pci_enable_msi &
request_irq
>>> >
>>> > What shows up in the Xen console when you send the
''q'' key? Does it
>>> > show that the vector is assigned to the appropiate guest?
>>>
>>> The Xen console q key shows that the domU is assigned:
>>>
>>> (XEN)     Interrupts { 32, 41-42, 47 }
>>
>> Aha!
>>
>>>
>>> but the domU thinks it has:
>>>
>>> 124/125/126/127
>>>
>>> Is there some mapping that''s taking place, or is this
plain wrong?
>>
>> That looks wrong. The IRQ numbers (even though they are MSI vectors)
are
>> setup as IRQ numbers in the DomU guest. You should have seen
>>
>> 32:
>> 41:
>> 42:
>> 47:
>> in you /proc/interrupts on your DomU guest.
>>
>> I wonder what broke  - can you use
>git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git>
>> devel/xen-pcifront-0.5 (or pv/pcifront-2.6.32)?
>
>Please forgive the git ignorance.
>
>Is this the right syntax?
>
>git clone
git://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.32<http://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.32>
>linux-2.6.32-pv-pcifront
>
>Initialized empty Git repository in
>/import/kaan/bedge/src/xen/kernel/pv-ops/linux-2.6.32-pv-pcifront/.git/
>fatal: The remote end hung up unexpectedly
>
>Or:
>
> git clone 
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git>
>
>Initialized empty Git repository in
>/import/kaan/bedge/src/xen/kernel/pv-ops/xen/.git/
>remote: error: Could not read 59eab2f8f04147c5aadc99f2034ca7e5b81e890f
>remote: fatal: Failed to traverse parents of commit
>979e121cb348add17ed8171bf447b27a3a9d1be3
>remote: aborting due to possible repository corruption on the remote side.
>fatal: early EOF
>fatal: index-pack failed
>
>>
>> It has the latest pcifront driver but without the PVonHVM enhancments
>> so we can try to eliminate the PvONHVM logic out of the picture.
>>
>>>
>>> >
>>> > > - the interrupt does happen. But the interrupt service
routine of
>>> > > tachyon driver doesn''t detect any interrupt
status related to this
>>> > > interrupt, which inhibits the tachyon chip from coming
on-line. And
>>> > > there are high count of tachyon interrupt in
/proc/interrupts
>>> >
>>> > Is it checking the PCI_STATUS_INTERRUPT or the appropiate
register
>>> > in the MMIO BAR?
>>> >
>>>
>>> The driver would check the appropriate register (tachyon registers)
in
>>> the MMIO to determine the source of interrupts.
>>
>> OK, so that isn''t it. Is there anything at these vectors:
>> 7c, 7d, 7e, and 7f? When you use xen debug-keys ''i''
or ''q'' it should give you
>> an inkling what device this is set for.
>
>When I run a distro kernel in hvm mode, I get the expected irq mappings:
>
>''i'' - Note 66 - 69
>(XEN)    IRQ:  66 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:3a
>type=PCI-MSI         status=00000010 in-flight=0
>domain-list=10:127(----),
>(XEN)    IRQ:  67 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:42
>type=PCI-MSI         status=00000010 in-flight=0
>domain-list=10:126(----),
>(XEN)    IRQ:  68 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:4a
>type=PCI-MSI         status=00000010 in-flight=0
>domain-list=10:125(----),
>(XEN)    IRQ:  69 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:52
>type=PCI-MSI         status=00000010 in-flight=0
>domain-list=10:124(----)
>
>
>''q''
>(XEN)     Interrupts { 32, 41-42, 47, 124-127 }
>
>
>The same data with pv-ops kernel shows:
>
>''i''
>IRQ numbers stop at 65, no 66 - 69 present:
>
>(XEN)    IRQ:  63 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:91
>type=PCI-MSI         status=00000010 in-flight=0
>domain-list=0:289(----),
>(XEN)    IRQ:  64 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:99
>type=PCI-MSI         status=00000002 mapped, unbound
>(XEN)    IRQ:  65 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:b1
>type=PCI-MSI         status=00000010 in-flight=0
>domain-list=0:287(----),
>(XEN) IO-APIC interrupt information:
>
>''q''
>(XEN)     Interrupts { 32, 41-42, 47 }
>
>>
>>>
>>> > >
>>> > > kaan-18-dpm:~# cat /proc/interrupts | grep TACH
>>> > >
>124:     760415          0          0          0          0
>    0
>>> > >          0          0          0          0          0
>      0
>>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
>>> > >
>125:     762234          0          0          0          0
>    0
>>> > >          0          0          0          0          0
>      0
>>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
>>> > >
>126:     764180          0          0          0          0
>    0
>>> > >          0          0          0          0          0
>      0
>>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
>>> > >
>127:     764164          0          0          0          0
>    0
>>> > >          0          0          0          0          0
>      0
>>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
>>> >
>>> > Can you provide the full dmesg output?
>>>
>>> Attached.
>>>
>>> Some possibly related messages on dom0 console:
>>>
>>> [ 1882.269778] pciback 0000:07:00.0: enabling device (0000 ->
0003)
>>> [ 1882.269800] xen: registering gsi 32 triggering 0 polarity 1
>>> [ 1882.269827] xen_allocate_pirq: returning irq 32 for gsi 32
>>> [ 1882.269834] xen: --> irq=32
>>> [ 1882.269841] Already setup the GSI :32
>>> [ 1882.269847] pciback 0000:07:00.0: PCI INT A -> GSI 32 (level,
low) -> IRQ 32
>>> [ 1882.269866] pciback 0000:07:00.0: setting latency timer to 64
>>> [ 1882.270463] pciback 0000:07:00.0: Driver tried to write to a
>>> read-only configuration space field at offset 0x62, size 2. This
may
>>> be harmless, but if you have problems with your device:
>>
>> Uhhh, for that I think you need to do ''lspci -vvv -xxx -s
07:00.00''
>> to find out what is at the configuration space. You could enable
>> it using the permissive attribute.
>>
>>> [ 1882.270465] 1) see permissive attribute in sysfs
>>> [ 1882.270467] 2) report problems to the xen-devel mailing list
along
>>> with details of your device obtained from lspci.
>>> [ 1882.270615]   alloc irq_desc for 478 on node 0
>>> [ 1882.270625]   alloc kstat_irqs on node 0
>>
>> So for 478: what do you see? xen-pciback I presume?
>>> [ 1882.348411] pciback 0000:07:00.1: enabling device (0000 ->
0003)
>>> [ 1882.348433] xen: registering gsi 42 triggering 0 polarity 1
>>> [ 1882.348440] xen_allocate_pirq: returning irq 42 for gsi 42
>>> [ 1882.348445] xen: --> irq=42
>>> [ 1882.348472] Already setup the GSI :42
>>> [ 1882.348479] pciback 0000:07:00.1: PCI INT B -> GSI 42 (level,
low) -> IRQ 42
>>> [ 1882.348497] pciback 0000:07:00.1: setting latency timer to 64
>>> [ 1882.349063] pciback 0000:07:00.1: Driver tried to write to a
>>> read-only configuration space field at offset 0x62, size 2. This
may
>>> be harmless, but if you have problems with your device:
>>> [ 1882.349066] 1) see permissive attribute in sysfs
>>> [ 1882.349067] 2) report problems to the xen-devel mailing list
along
>>> with details of your device obtained from lspci.
>>> [ 1882.349205]   alloc irq_desc for 477 on node 0
>>> [ 1882.349215]   alloc kstat_irqs on node 0
>>> [ 1882.402893] pciback 0000:07:00.2: enabling device (0000 ->
0003)
>>> [ 1882.402908] xen: registering gsi 47 triggering 0 polarity 1
>>> [ 1882.402913] xen_allocate_pirq: returning irq 47 for gsi 47
>>> [ 1882.402916] xen: --> irq=47
>>> [ 1882.402921] Already setup the GSI :47
>>> [ 1882.402925] pciback 0000:07:00.2: PCI INT C -> GSI 47 (level,
low) -> IRQ 47
>>> [ 1882.402938] pciback 0000:07:00.2: setting latency timer to 64
>>> [ 1882.403280] pciback 0000:07:00.2: Driver tried to write to a
>>> read-only configuration space field at offset 0x62, size 2. This
may
>>> be harmless, but if you have problems with your device:
>>> [ 1882.403282] 1) see permissive attribute in sysfs
>>> [ 1882.403282] 2) report problems to the xen-devel mailing list
along
>>> with details of your device obtained from lspci.
>>> [ 1882.403380]   alloc irq_desc for 476 on node 0
>>> [ 1882.403386]   alloc kstat_irqs on node 0
>>> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
>>> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
>>> fault addr e6f80000, iommu reg = ffff82c3fff57000
>>> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
>>> (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn
= e6f80
>>> (XEN)     root_entry = ffff83019ff70000
>>> (XEN)     root_entry[7] = 19cf52001
>>> (XEN)     context = ffff83019cf52000
>>> (XEN)     context[0] = 102_706dc005
>>> (XEN)     l4 = ffff8300706dc000
>>> (XEN)     l4_index = 0
>>> (XEN)     l4[0] = 706db003
>>> (XEN)     l3 = ffff8300706db000
>>> (XEN)     l3_index = 3
>>> (XEN)     l3[3] = 702b6003
>>> (XEN)     l2 = ffff8300702b6000
>>> (XEN)     l2_index = 137
>>> (XEN)     l2[137] = 0
>>> (XEN)     l2[137] not present
>>> (XEN) traps.c:466:d0 Unhandled nmi fault/trap [#2] on VCPU 0
[ec=0000]
>>
>> That is not good. What changed from your earlier emails that this was
triggered?
>
>Nothing
>> Or was it triggered all along?
>
>Yes, I just included it for completeness
>
>> What happens if you run the system without the iommu enabled?
>
>Haven''t tried yet. Will check that next.
>
>-Bruce
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com<mailto:Xen-devel@lists.xensource.com>
>http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Bruce Edge

2010-Sep-28 04:45 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Mon, Sep 27, 2010 at 8:26 PM, Jiang, Yunhong
<yunhong.jiang@intel.com>wrote:
>  "xm dmesg" should gives xen''s boot log, and sometimes
it contain some
> helpful information, I think, especially loglvl and guest_loglvl is set to
> all.
>
>I looked at the xm dmesg output and there''s nothing more than what I
already
provided, aside from a bunch of commands from me poking at it.

-Bruce

>
>
> Thanks
>
> --jyh
>
>
>
> *From:* Bruce Edge [mailto:bruce.edge@gmail.com]
> *Sent:* Tuesday, September 28, 2010 11:16 AM
> *To:* Jiang, Yunhong
> *Cc:* Konrad Rzeszutek Wilk; xen-devel@lists.xensource.com
>
> *Subject:* Re: [Xen-devel] pv-ops domU not working with MSI interrupts on
> Nehalem
>
>
>
> On Mon, Sep 27, 2010 at 6:15 PM, Jiang, Yunhong
<yunhong.jiang@intel.com>
> wrote:
>
> Is the 07:0.0 your tachyon device? The VT-d fault is suspcious.
>
>
>
> Yes, there is 1 quad port card is this sytem:
>
>
>
> 07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
>
> 07:00.1 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
>
> 07:00.2 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
>
> 07:00.3 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
>
>
>
>
>
> Also is it possible to share the xen output?
>
>
>
> I attached the dom0 boot output. Let me know if you wanted something else.
>
>
>
> Also, here''s the dom0 console output upon starting the VM: This
lockdep
> error started with the release of 2.6.32.21. Note that I''m running
the same
>  kernel for the domU and dom0.
>
>
>
> [ 1817.684097] ------------[ cut here ]------------
>
> [ 1817.684113] WARNING: at kernel/lockdep.c:2323
> trace_hardirqs_on_caller+0x12f/0x190()
>
> [ 1817.684119] Hardware name: ProLiant DL380 G6
>
> [ 1817.684122] Modules linked in: xt_physdev ipv6 osa_mfgdom0 xenfs
> xen_gntdev fbcon tileblit font bitblit softcursor xen_evtchn xen_pciback
> radeon ttm drm_kms_helper tun drm i2c_algo_bit ipmi_si i2c_core
> ipmi_msghandler joydev serio_raw hpwdt hpilo bridge stp llc usbhid hid
cciss
> usb_storage
>
> [ 1817.684190] Pid: 11, comm: xenwatch Not tainted 2.6.32.21-xenoprof-1 #1
>
> [ 1817.684195] Call Trace:
>
> [ 1817.684197]  <IRQ>  [<ffffffff810aa18f>] ?
> trace_hardirqs_on_caller+0x12f/0x190
>
> [ 1817.684209]  [<ffffffff8106bed0>] warn_slowpath_common+0x80/0xd0
>
> [ 1817.684217]  [<ffffffff815f2b80>] ? _spin_unlock_irq+0x30/0x40
>
> [ 1817.684223]  [<ffffffff8106bf34>] warn_slowpath_null+0x14/0x20
>
> [ 1817.684229]  [<ffffffff810aa18f>]
trace_hardirqs_on_caller+0x12f/0x190
>
> [ 1817.684234]  [<ffffffff810aa1fd>] trace_hardirqs_on+0xd/0x10
>
> [ 1817.684240]  [<ffffffff815f2b80>] _spin_unlock_irq+0x30/0x40
>
> [ 1817.684266]  [<ffffffff813c4fc5>]
> add_to_net_schedule_list_tail+0x85/0xd0
>
> [ 1817.684271]  [<ffffffff813c6216>] netif_be_int+0x36/0x160
>
> [ 1817.684278]  [<ffffffff810e10d0>] handle_IRQ_event+0x70/0x180
>
> [ 1817.684284]  [<ffffffff810e36e9>] handle_edge_irq+0xc9/0x170
>
> [ 1817.684291]  [<ffffffff813b8d7f>]
__xen_evtchn_do_upcall+0x1bf/0x1f0
>
> [ 1817.684297]  [<ffffffff813b92fd>] xen_evtchn_do_upcall+0x3d/0x60
>
> [ 1817.684304]  [<ffffffff8101647e>]
xen_do_hypervisor_callback+0x1e/0x30
>
> [ 1817.684308]  <EOI>  [<ffffffff8100940a>] ?
hypercall_page+0x40a/0x1010
>
> [ 1817.684319]  [<ffffffff8100940a>] ? hypercall_page+0x40a/0x1010
>
> [ 1817.684325]  [<ffffffff813bce54>] ? xb_write+0x1e4/0x290
>
> [ 1817.684330]  [<ffffffff813bd8ca>] ? xs_talkv+0x6a/0x1f0
>
> [ 1817.684336]  [<ffffffff813bd8d8>] ? xs_talkv+0x78/0x1f0
>
> [ 1817.684341]  [<ffffffff813bdbcd>] ? xs_single+0x4d/0x60
>
> [ 1817.684346]  [<ffffffff813be502>] ? xenbus_read+0x52/0x80
>
> [ 1817.684352]  [<ffffffff813c87fc>] ? frontend_changed+0x48c/0x770
>
> [ 1817.684358]  [<ffffffff813bf76d>] ?
xenbus_otherend_changed+0xdd/0x1b0
>
> [ 1817.684365]  [<ffffffff8101122f>] ?
xen_restore_fl_direct_end+0x0/0x1
>
> [ 1817.684371]  [<ffffffff810ac830>] ? lock_release+0xb0/0x230
>
> [ 1817.684376]  [<ffffffff813bfae0>] ? frontend_changed+0x10/0x20
>
> [ 1817.684382]  [<ffffffff813bd4f5>] ? xenwatch_thread+0x55/0x160
>
> [ 1817.684389]  [<ffffffff81093400>] ?
autoremove_wake_function+0x0/0x40
>
> [ 1817.684394]  [<ffffffff813bd4a0>] ? xenwatch_thread+0x0/0x160
>
> [ 1817.684400]  [<ffffffff81093086>] ? kthread+0x96/0xb0
>
> [ 1817.684405]  [<ffffffff8101632a>] ? child_rip+0xa/0x20
>
> [ 1817.684410]  [<ffffffff81015c90>] ? restore_args+0x0/0x30
>
> [ 1817.684415]  [<ffffffff81016320>] ? child_rip+0x0/0x20
>
>
>
> -Bruce
>
>
>
>
>
>
> Thanks
> --jyh
>
>
> >-----Original Message-----
> >From: xen-devel-bounces@lists.xensource.com
> >[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Bruce Edge
> >Sent: Tuesday, September 28, 2010 7:54 AM
> >To: Konrad Rzeszutek Wilk
> >Cc: xen-devel@lists.xensource.com
>
> >Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on
> Nehalem
> >
>
> >On Mon, Sep 27, 2010 at 12:54 PM, Konrad Rzeszutek Wilk
> ><konrad.wilk@oracle.com> wrote:
> >> On Mon, Sep 27, 2010 at 12:16:50PM -0700, Bruce Edge wrote:
> >>> On Mon, Sep 27, 2010 at 10:24 AM, Konrad Rzeszutek Wilk
> >>> <konrad.wilk@oracle.com> wrote:
> >>> >
> >>> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge
wrote:
> >>> > > One of our developers who is working on a tachyon
driver is
> >>> > > complaining that the pvops domU kernel is not
working for these MSI
> >>> > > interrupts.
> >>> > > This is using the current head of xen/2.6.32.x on
both a single
> >>> > > Nahelam 920 and a dual E5540. This behavior is
consistent with Xen
> >>> > > 4.0.1, 4.0.2.rc1-pre and 4.1.
> >>> > >
> >>> > > Here are his comments:
> >>> > >
> >>> > > - the driver has no problem to enable msi interrupt
and request the
> >>> > > interrupt through kernel functions pci_enable_msi
& request_irq
> >>> >
> >>> > What shows up in the Xen console when you send the
''q'' key? Does it
> >>> > show that the vector is assigned to the appropiate guest?
> >>>
> >>> The Xen console q key shows that the domU is assigned:
> >>>
> >>> (XEN)     Interrupts { 32, 41-42, 47 }
> >>
> >> Aha!
> >>
> >>>
> >>> but the domU thinks it has:
> >>>
> >>> 124/125/126/127
> >>>
> >>> Is there some mapping that''s taking place, or is this
plain wrong?
> >>
> >> That looks wrong. The IRQ numbers (even though they are MSI
vectors) are
> >> setup as IRQ numbers in the DomU guest. You should have seen
> >>
> >> 32:
> >> 41:
> >> 42:
> >> 47:
> >> in you /proc/interrupts on your DomU guest.
> >>
> >> I wonder what broke  - can you use
> >git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
> >> devel/xen-pcifront-0.5 (or pv/pcifront-2.6.32)?
> >
> >Please forgive the git ignorance.
> >
> >Is this the right syntax?
> >
> >git clone git://
> git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.32
> >linux-2.6.32-pv-pcifront
> >
> >Initialized empty Git repository in
> >/import/kaan/bedge/src/xen/kernel/pv-ops/linux-2.6.32-pv-pcifront/.git/
> >fatal: The remote end hung up unexpectedly
> >
> >Or:
> >
> > git clone 
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
> >
> >Initialized empty Git repository in
> >/import/kaan/bedge/src/xen/kernel/pv-ops/xen/.git/
> >remote: error: Could not read 59eab2f8f04147c5aadc99f2034ca7e5b81e890f
> >remote: fatal: Failed to traverse parents of commit
> >979e121cb348add17ed8171bf447b27a3a9d1be3
> >remote: aborting due to possible repository corruption on the remote
side.
> >fatal: early EOF
> >fatal: index-pack failed
> >
> >>
> >> It has the latest pcifront driver but without the PVonHVM
enhancments
> >> so we can try to eliminate the PvONHVM logic out of the picture.
> >>
> >>>
> >>> >
> >>> > > - the interrupt does happen. But the interrupt
service routine of
> >>> > > tachyon driver doesn''t detect any interrupt
status related to this
> >>> > > interrupt, which inhibits the tachyon chip from
coming on-line. And
> >>> > > there are high count of tachyon interrupt in
/proc/interrupts
> >>> >
> >>> > Is it checking the PCI_STATUS_INTERRUPT or the appropiate
register
> >>> > in the MMIO BAR?
> >>> >
> >>>
> >>> The driver would check the appropriate register (tachyon
registers) in
> >>> the MMIO to determine the source of interrupts.
> >>
> >> OK, so that isn''t it. Is there anything at these vectors:
> >> 7c, 7d, 7e, and 7f? When you use xen debug-keys
''i'' or ''q'' it should
> give you
> >> an inkling what device this is set for.
> >
> >When I run a distro kernel in hvm mode, I get the expected irq
mappings:
> >
> >''i'' - Note 66 - 69
> >(XEN)    IRQ:  66 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:3a
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=10:127(----),
> >(XEN)    IRQ:  67 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:42
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=10:126(----),
> >(XEN)    IRQ:  68 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:4a
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=10:125(----),
> >(XEN)    IRQ:  69 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:52
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=10:124(----)
> >
> >
> >''q''
> >(XEN)     Interrupts { 32, 41-42, 47, 124-127 }
> >
> >
> >The same data with pv-ops kernel shows:
> >
> >''i''
> >IRQ numbers stop at 65, no 66 - 69 present:
> >
> >(XEN)    IRQ:  63 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:91
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=0:289(----),
> >(XEN)    IRQ:  64 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:99
> >type=PCI-MSI         status=00000002 mapped, unbound
> >(XEN)    IRQ:  65 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:b1
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=0:287(----),
> >(XEN) IO-APIC interrupt information:
> >
> >''q''
> >(XEN)     Interrupts { 32, 41-42, 47 }
> >
> >>
> >>>
> >>> > >
> >>> > > kaan-18-dpm:~# cat /proc/interrupts | grep TACH
> >>> > >
> >124:     760415          0          0          0          0
> >    0
> >>> > >          0          0          0          0         
0
> >      0
> >>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> >>> > >
> >125:     762234          0          0          0          0
> >    0
> >>> > >          0          0          0          0         
0
> >      0
> >>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> >>> > >
> >126:     764180          0          0          0          0
> >    0
> >>> > >          0          0          0          0         
0
> >      0
> >>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> >>> > >
> >127:     764164          0          0          0          0
> >    0
> >>> > >          0          0          0          0         
0
> >      0
> >>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> >>> >
> >>> > Can you provide the full dmesg output?
> >>>
> >>> Attached.
> >>>
> >>> Some possibly related messages on dom0 console:
> >>>
> >>> [ 1882.269778] pciback 0000:07:00.0: enabling device (0000
-> 0003)
> >>> [ 1882.269800] xen: registering gsi 32 triggering 0 polarity 1
> >>> [ 1882.269827] xen_allocate_pirq: returning irq 32 for gsi 32
> >>> [ 1882.269834] xen: --> irq=32
> >>> [ 1882.269841] Already setup the GSI :32
> >>> [ 1882.269847] pciback 0000:07:00.0: PCI INT A -> GSI 32
(level, low)
> -> IRQ 32
> >>> [ 1882.269866] pciback 0000:07:00.0: setting latency timer to
64
> >>> [ 1882.270463] pciback 0000:07:00.0: Driver tried to write to
a
> >>> read-only configuration space field at offset 0x62, size 2.
This may
> >>> be harmless, but if you have problems with your device:
> >>
> >> Uhhh, for that I think you need to do ''lspci -vvv -xxx -s
07:00.00''
> >> to find out what is at the configuration space. You could enable
> >> it using the permissive attribute.
> >>
> >>> [ 1882.270465] 1) see permissive attribute in sysfs
> >>> [ 1882.270467] 2) report problems to the xen-devel mailing
list along
> >>> with details of your device obtained from lspci.
> >>> [ 1882.270615]   alloc irq_desc for 478 on node 0
> >>> [ 1882.270625]   alloc kstat_irqs on node 0
> >>
> >> So for 478: what do you see? xen-pciback I presume?
> >>> [ 1882.348411] pciback 0000:07:00.1: enabling device (0000
-> 0003)
> >>> [ 1882.348433] xen: registering gsi 42 triggering 0 polarity 1
> >>> [ 1882.348440] xen_allocate_pirq: returning irq 42 for gsi 42
> >>> [ 1882.348445] xen: --> irq=42
> >>> [ 1882.348472] Already setup the GSI :42
> >>> [ 1882.348479] pciback 0000:07:00.1: PCI INT B -> GSI 42
(level, low)
> -> IRQ 42
> >>> [ 1882.348497] pciback 0000:07:00.1: setting latency timer to
64
> >>> [ 1882.349063] pciback 0000:07:00.1: Driver tried to write to
a
> >>> read-only configuration space field at offset 0x62, size 2.
This may
> >>> be harmless, but if you have problems with your device:
> >>> [ 1882.349066] 1) see permissive attribute in sysfs
> >>> [ 1882.349067] 2) report problems to the xen-devel mailing
list along
> >>> with details of your device obtained from lspci.
> >>> [ 1882.349205]   alloc irq_desc for 477 on node 0
> >>> [ 1882.349215]   alloc kstat_irqs on node 0
> >>> [ 1882.402893] pciback 0000:07:00.2: enabling device (0000
-> 0003)
> >>> [ 1882.402908] xen: registering gsi 47 triggering 0 polarity 1
> >>> [ 1882.402913] xen_allocate_pirq: returning irq 47 for gsi 47
> >>> [ 1882.402916] xen: --> irq=47
> >>> [ 1882.402921] Already setup the GSI :47
> >>> [ 1882.402925] pciback 0000:07:00.2: PCI INT C -> GSI 47
(level, low)
> -> IRQ 47
> >>> [ 1882.402938] pciback 0000:07:00.2: setting latency timer to
64
> >>> [ 1882.403280] pciback 0000:07:00.2: Driver tried to write to
a
> >>> read-only configuration space field at offset 0x62, size 2.
This may
> >>> be harmless, but if you have problems with your device:
> >>> [ 1882.403282] 1) see permissive attribute in sysfs
> >>> [ 1882.403282] 2) report problems to the xen-devel mailing
list along
> >>> with details of your device obtained from lspci.
> >>> [ 1882.403380]   alloc irq_desc for 476 on node 0
> >>> [ 1882.403386]   alloc kstat_irqs on node 0
> >>> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending
Fault
> >>> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device
[07:00.0]
> >>> fault addr e6f80000, iommu reg = ffff82c3fff57000
> >>> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
> >>> (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0
gmfn > e6f80
> >>> (XEN)     root_entry = ffff83019ff70000
> >>> (XEN)     root_entry[7] = 19cf52001
> >>> (XEN)     context = ffff83019cf52000
> >>> (XEN)     context[0] = 102_706dc005
> >>> (XEN)     l4 = ffff8300706dc000
> >>> (XEN)     l4_index = 0
> >>> (XEN)     l4[0] = 706db003
> >>> (XEN)     l3 = ffff8300706db000
> >>> (XEN)     l3_index = 3
> >>> (XEN)     l3[3] = 702b6003
> >>> (XEN)     l2 = ffff8300702b6000
> >>> (XEN)     l2_index = 137
> >>> (XEN)     l2[137] = 0
> >>> (XEN)     l2[137] not present
> >>> (XEN) traps.c:466:d0 Unhandled nmi fault/trap [#2] on VCPU 0
[ec=0000]
> >>
> >> That is not good. What changed from your earlier emails that this
was
> triggered?
> >
> >Nothing
> >> Or was it triggered all along?
> >
> >Yes, I just included it for completeness
> >
> >> What happens if you run the system without the iommu enabled?
> >
> >Haven''t tried yet. Will check that next.
> >
> >-Bruce
> >
>
> >_______________________________________________
> >Xen-devel mailing list
> >Xen-devel@lists.xensource.com
> >http://lists.xensource.com/xen-devel
>
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk

2010-Sep-28 14:56 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

>  git clone  git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
> 
> Initialized empty Git repository in
> /import/kaan/bedge/src/xen/kernel/pv-ops/xen/.git/
> remote: error: Could not read 59eab2f8f04147c5aadc99f2034ca7e5b81e890f
> remote: fatal: Failed to traverse parents of commit
> 979e121cb348add17ed8171bf447b27a3a9d1be3
> remote: aborting due to possible repository corruption on the remote side.
> fatal: early EOF
> fatal: index-pack failed
That should have worked, but it looks as my git repo is busted. Let me fix that
and once it done you should be able to do

cd xen
git checkout origin/pv/pcifront-2.6.32
> 
> >
> > It has the latest pcifront driver but without the PVonHVM enhancments
> > so we can try to eliminate the PvONHVM logic out of the picture.
> >
> >>
> >> >
> >> > > - the interrupt does happen. But the interrupt service
routine of
> >> > > tachyon driver doesn''t detect any interrupt
status related to this
> >> > > interrupt, which inhibits the tachyon chip from coming
on-line. And
> >> > > there are high count of tachyon interrupt in
/proc/interrupts
> >> >
> >> > Is it checking the PCI_STATUS_INTERRUPT or the appropiate
register
> >> > in the MMIO BAR?
> >> >
> >>
> >> The driver would check the appropriate register (tachyon
registers) in
> >> the MMIO to determine the source of interrupts.
> >
> > OK, so that isn''t it. Is there anything at these vectors:
> > 7c, 7d, 7e, and 7f? When you use xen debug-keys ''i''
or ''q'' it should give you
> > an inkling what device this is set for.
> 
> When I run a distro kernel in hvm mode, I get the expected irq mappings:
> 
> ''i'' - Note 66 - 69
> (XEN)    IRQ:  66 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:3a
> type=PCI-MSI         status=00000010 in-flight=0
> domain-list=10:127(----),
> (XEN)    IRQ:  67 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:42
> type=PCI-MSI         status=00000010 in-flight=0
> domain-list=10:126(----),
> (XEN)    IRQ:  68 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:4a
> type=PCI-MSI         status=00000010 in-flight=0
> domain-list=10:125(----),
> (XEN)    IRQ:  69 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:52
> type=PCI-MSI         status=00000010 in-flight=0
> domain-list=10:124(----)
> 
What does cat /proc/interrupts (don''t do the grep) for this HVM
guest?> 
> ''q''
> (XEN)     Interrupts { 32, 41-42, 47, 124-127 }
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Lin, Ray

2010-Sep-28 16:08 UTC

head link

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

I just checked the "xen dmesg". Look like DMA/iommu is the root cause
of this issue. In order to tell the source of interrupt, Tachyon chip needs to
do the DMA write to a dword memory location to indicate the source of interrupt.
What iommu option do you recommend to use ?

(XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0] fault addr
c00000, iommu reg = ffff82c3fff57000
(XEN) DMAR:[fault reason 05h] PTE Write access is not set
(XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
(XEN)     root_entry = ffff83019ff70000
(XEN)     root_entry[7] = 19cf52001
(XEN)     context = ffff83019cf52000
(XEN)     context[0] = 102_706dc005
(XEN)     l4 = ffff8300706dc000
(XEN)     l4_index = 0
(XEN)     l4[0] = 706db003
(XEN)     l3 = ffff8300706db000
(XEN)     l3_index = 0
(XEN)     l3[0] = 706da003
(XEN)     l2 = ffff8300706da000
(XEN)     l2_index = 6
(XEN)     l2[6] = 0


-Ray


________________________________
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Bruce Edge
Sent: Monday, September 27, 2010 9:46 PM
To: Jiang, Yunhong
Cc: xen-devel@lists.xensource.com; Konrad Rzeszutek Wilk
Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Mon, Sep 27, 2010 at 8:26 PM, Jiang, Yunhong
<yunhong.jiang@intel.com<mailto:yunhong.jiang@intel.com>> wrote:
"xm dmesg" should gives xen''s boot log, and sometimes it
contain some helpful information, I think, especially loglvl and guest_loglvl is
set to all.

I looked at the xm dmesg output and there''s nothing more than what I
already provided, aside from a bunch of commands from me poking at it.

-Bruce


Thanks
--jyh

From: Bruce Edge
[mailto:bruce.edge@gmail.com<mailto:bruce.edge@gmail.com>]
Sent: Tuesday, September 28, 2010 11:16 AM
To: Jiang, Yunhong
Cc: Konrad Rzeszutek Wilk;
xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>

Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Mon, Sep 27, 2010 at 6:15 PM, Jiang, Yunhong
<yunhong.jiang@intel.com<mailto:yunhong.jiang@intel.com>> wrote:
Is the 07:0.0 your tachyon device? The VT-d fault is suspcious.

Yes, there is 1 quad port card is this sytem:

07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
07:00.1 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
07:00.2 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
07:00.3 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)


Also is it possible to share the xen output?

I attached the dom0 boot output. Let me know if you wanted something else.

Also, here''s the dom0 console output upon starting the VM: This lockdep
error started with the release of 2.6.32.21. Note that I''m running the
same  kernel for the domU and dom0.

[ 1817.684097] ------------[ cut here ]------------
[ 1817.684113] WARNING: at kernel/lockdep.c:2323
trace_hardirqs_on_caller+0x12f/0x190()
[ 1817.684119] Hardware name: ProLiant DL380 G6
[ 1817.684122] Modules linked in: xt_physdev ipv6 osa_mfgdom0 xenfs xen_gntdev
fbcon tileblit font bitblit softcursor xen_evtchn xen_pciback radeon ttm
drm_kms_helper tun drm i2c_algo_bit ipmi_si i2c_core ipmi_msghandler joydev
serio_raw hpwdt hpilo bridge stp llc usbhid hid cciss usb_storage
[ 1817.684190] Pid: 11, comm: xenwatch Not tainted 2.6.32.21-xenoprof-1 #1
[ 1817.684195] Call Trace:
[ 1817.684197]  <IRQ>  [<ffffffff810aa18f>] ?
trace_hardirqs_on_caller+0x12f/0x190
[ 1817.684209]  [<ffffffff8106bed0>] warn_slowpath_common+0x80/0xd0
[ 1817.684217]  [<ffffffff815f2b80>] ? _spin_unlock_irq+0x30/0x40
[ 1817.684223]  [<ffffffff8106bf34>] warn_slowpath_null+0x14/0x20
[ 1817.684229]  [<ffffffff810aa18f>] trace_hardirqs_on_caller+0x12f/0x190
[ 1817.684234]  [<ffffffff810aa1fd>] trace_hardirqs_on+0xd/0x10
[ 1817.684240]  [<ffffffff815f2b80>] _spin_unlock_irq+0x30/0x40
[ 1817.684266]  [<ffffffff813c4fc5>]
add_to_net_schedule_list_tail+0x85/0xd0
[ 1817.684271]  [<ffffffff813c6216>] netif_be_int+0x36/0x160
[ 1817.684278]  [<ffffffff810e10d0>] handle_IRQ_event+0x70/0x180
[ 1817.684284]  [<ffffffff810e36e9>] handle_edge_irq+0xc9/0x170
[ 1817.684291]  [<ffffffff813b8d7f>] __xen_evtchn_do_upcall+0x1bf/0x1f0
[ 1817.684297]  [<ffffffff813b92fd>] xen_evtchn_do_upcall+0x3d/0x60
[ 1817.684304]  [<ffffffff8101647e>] xen_do_hypervisor_callback+0x1e/0x30
[ 1817.684308]  <EOI>  [<ffffffff8100940a>] ?
hypercall_page+0x40a/0x1010
[ 1817.684319]  [<ffffffff8100940a>] ? hypercall_page+0x40a/0x1010
[ 1817.684325]  [<ffffffff813bce54>] ? xb_write+0x1e4/0x290
[ 1817.684330]  [<ffffffff813bd8ca>] ? xs_talkv+0x6a/0x1f0
[ 1817.684336]  [<ffffffff813bd8d8>] ? xs_talkv+0x78/0x1f0
[ 1817.684341]  [<ffffffff813bdbcd>] ? xs_single+0x4d/0x60
[ 1817.684346]  [<ffffffff813be502>] ? xenbus_read+0x52/0x80
[ 1817.684352]  [<ffffffff813c87fc>] ? frontend_changed+0x48c/0x770
[ 1817.684358]  [<ffffffff813bf76d>] ? xenbus_otherend_changed+0xdd/0x1b0
[ 1817.684365]  [<ffffffff8101122f>] ? xen_restore_fl_direct_end+0x0/0x1
[ 1817.684371]  [<ffffffff810ac830>] ? lock_release+0xb0/0x230
[ 1817.684376]  [<ffffffff813bfae0>] ? frontend_changed+0x10/0x20
[ 1817.684382]  [<ffffffff813bd4f5>] ? xenwatch_thread+0x55/0x160
[ 1817.684389]  [<ffffffff81093400>] ? autoremove_wake_function+0x0/0x40
[ 1817.684394]  [<ffffffff813bd4a0>] ? xenwatch_thread+0x0/0x160
[ 1817.684400]  [<ffffffff81093086>] ? kthread+0x96/0xb0
[ 1817.684405]  [<ffffffff8101632a>] ? child_rip+0xa/0x20
[ 1817.684410]  [<ffffffff81015c90>] ? restore_args+0x0/0x30
[ 1817.684415]  [<ffffffff81016320>] ? child_rip+0x0/0x20

-Bruce



Thanks
--jyh
>-----Original Message-----
>From:
xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounces@lists.xensource.com>
>[mailto:xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounces@lists.xensource.com>]
On Behalf Of Bruce Edge
>Sent: Tuesday, September 28, 2010 7:54 AM
>To: Konrad Rzeszutek Wilk
>Cc:
xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>
>Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on
Nehalem
>
>On Mon, Sep 27, 2010 at 12:54 PM, Konrad Rzeszutek Wilk
><konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>> wrote:
>> On Mon, Sep 27, 2010 at 12:16:50PM -0700, Bruce Edge wrote:
>>> On Mon, Sep 27, 2010 at 10:24 AM, Konrad Rzeszutek Wilk
>>> <konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>>
wrote:
>>> >
>>> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge wrote:
>>> > > One of our developers who is working on a tachyon driver
is
>>> > > complaining that the pvops domU kernel is not working for
these MSI
>>> > > interrupts.
>>> > > This is using the current head of xen/2.6.32.x on both a
single
>>> > > Nahelam 920 and a dual E5540. This behavior is consistent
with Xen
>>> > > 4.0.1, 4.0.2.rc1-pre and 4.1.
>>> > >
>>> > > Here are his comments:
>>> > >
>>> > > - the driver has no problem to enable msi interrupt and
request the
>>> > > interrupt through kernel functions pci_enable_msi &
request_irq
>>> >
>>> > What shows up in the Xen console when you send the
''q'' key? Does it
>>> > show that the vector is assigned to the appropiate guest?
>>>
>>> The Xen console q key shows that the domU is assigned:
>>>
>>> (XEN)     Interrupts { 32, 41-42, 47 }
>>
>> Aha!
>>
>>>
>>> but the domU thinks it has:
>>>
>>> 124/125/126/127
>>>
>>> Is there some mapping that''s taking place, or is this
plain wrong?
>>
>> That looks wrong. The IRQ numbers (even though they are MSI vectors)
are
>> setup as IRQ numbers in the DomU guest. You should have seen
>>
>> 32:
>> 41:
>> 42:
>> 47:
>> in you /proc/interrupts on your DomU guest.
>>
>> I wonder what broke  - can you use
>git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git>
>> devel/xen-pcifront-0.5 (or pv/pcifront-2.6.32)?
>
>Please forgive the git ignorance.
>
>Is this the right syntax?
>
>git clone
git://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.32<http://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.32>
>linux-2.6.32-pv-pcifront
>
>Initialized empty Git repository in
>/import/kaan/bedge/src/xen/kernel/pv-ops/linux-2.6.32-pv-pcifront/.git/
>fatal: The remote end hung up unexpectedly
>
>Or:
>
> git clone 
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git>
>
>Initialized empty Git repository in
>/import/kaan/bedge/src/xen/kernel/pv-ops/xen/.git/
>remote: error: Could not read 59eab2f8f04147c5aadc99f2034ca7e5b81e890f
>remote: fatal: Failed to traverse parents of commit
>979e121cb348add17ed8171bf447b27a3a9d1be3
>remote: aborting due to possible repository corruption on the remote side.
>fatal: early EOF
>fatal: index-pack failed
>
>>
>> It has the latest pcifront driver but without the PVonHVM enhancments
>> so we can try to eliminate the PvONHVM logic out of the picture.
>>
>>>
>>> >
>>> > > - the interrupt does happen. But the interrupt service
routine of
>>> > > tachyon driver doesn''t detect any interrupt
status related to this
>>> > > interrupt, which inhibits the tachyon chip from coming
on-line. And
>>> > > there are high count of tachyon interrupt in
/proc/interrupts
>>> >
>>> > Is it checking the PCI_STATUS_INTERRUPT or the appropiate
register
>>> > in the MMIO BAR?
>>> >
>>>
>>> The driver would check the appropriate register (tachyon registers)
in
>>> the MMIO to determine the source of interrupts.
>>
>> OK, so that isn''t it. Is there anything at these vectors:
>> 7c, 7d, 7e, and 7f? When you use xen debug-keys ''i''
or ''q'' it should give you
>> an inkling what device this is set for.
>
>When I run a distro kernel in hvm mode, I get the expected irq mappings:
>
>''i'' - Note 66 - 69
>(XEN)    IRQ:  66 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:3a
>type=PCI-MSI         status=00000010 in-flight=0
>domain-list=10:127(----),
>(XEN)    IRQ:  67 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:42
>type=PCI-MSI         status=00000010 in-flight=0
>domain-list=10:126(----),
>(XEN)    IRQ:  68 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:4a
>type=PCI-MSI         status=00000010 in-flight=0
>domain-list=10:125(----),
>(XEN)    IRQ:  69 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:52
>type=PCI-MSI         status=00000010 in-flight=0
>domain-list=10:124(----)
>
>
>''q''
>(XEN)     Interrupts { 32, 41-42, 47, 124-127 }
>
>
>The same data with pv-ops kernel shows:
>
>''i''
>IRQ numbers stop at 65, no 66 - 69 present:
>
>(XEN)    IRQ:  63 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:91
>type=PCI-MSI         status=00000010 in-flight=0
>domain-list=0:289(----),
>(XEN)    IRQ:  64 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:99
>type=PCI-MSI         status=00000002 mapped, unbound
>(XEN)    IRQ:  65 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:b1
>type=PCI-MSI         status=00000010 in-flight=0
>domain-list=0:287(----),
>(XEN) IO-APIC interrupt information:
>
>''q''
>(XEN)     Interrupts { 32, 41-42, 47 }
>
>>
>>>
>>> > >
>>> > > kaan-18-dpm:~# cat /proc/interrupts | grep TACH
>>> > >
>124:     760415          0          0          0          0
>    0
>>> > >          0          0          0          0          0
>      0
>>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
>>> > >
>125:     762234          0          0          0          0
>    0
>>> > >          0          0          0          0          0
>      0
>>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
>>> > >
>126:     764180          0          0          0          0
>    0
>>> > >          0          0          0          0          0
>      0
>>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
>>> > >
>127:     764164          0          0          0          0
>    0
>>> > >          0          0          0          0          0
>      0
>>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
>>> >
>>> > Can you provide the full dmesg output?
>>>
>>> Attached.
>>>
>>> Some possibly related messages on dom0 console:
>>>
>>> [ 1882.269778] pciback 0000:07:00.0: enabling device (0000 ->
0003)
>>> [ 1882.269800] xen: registering gsi 32 triggering 0 polarity 1
>>> [ 1882.269827] xen_allocate_pirq: returning irq 32 for gsi 32
>>> [ 1882.269834] xen: --> irq=32
>>> [ 1882.269841] Already setup the GSI :32
>>> [ 1882.269847] pciback 0000:07:00.0: PCI INT A -> GSI 32 (level,
low) -> IRQ 32
>>> [ 1882.269866] pciback 0000:07:00.0: setting latency timer to 64
>>> [ 1882.270463] pciback 0000:07:00.0: Driver tried to write to a
>>> read-only configuration space field at offset 0x62, size 2. This
may
>>> be harmless, but if you have problems with your device:
>>
>> Uhhh, for that I think you need to do ''lspci -vvv -xxx -s
07:00.00''
>> to find out what is at the configuration space. You could enable
>> it using the permissive attribute.
>>
>>> [ 1882.270465] 1) see permissive attribute in sysfs
>>> [ 1882.270467] 2) report problems to the xen-devel mailing list
along
>>> with details of your device obtained from lspci.
>>> [ 1882.270615]   alloc irq_desc for 478 on node 0
>>> [ 1882.270625]   alloc kstat_irqs on node 0
>>
>> So for 478: what do you see? xen-pciback I presume?
>>> [ 1882.348411] pciback 0000:07:00.1: enabling device (0000 ->
0003)
>>> [ 1882.348433] xen: registering gsi 42 triggering 0 polarity 1
>>> [ 1882.348440] xen_allocate_pirq: returning irq 42 for gsi 42
>>> [ 1882.348445] xen: --> irq=42
>>> [ 1882.348472] Already setup the GSI :42
>>> [ 1882.348479] pciback 0000:07:00.1: PCI INT B -> GSI 42 (level,
low) -> IRQ 42
>>> [ 1882.348497] pciback 0000:07:00.1: setting latency timer to 64
>>> [ 1882.349063] pciback 0000:07:00.1: Driver tried to write to a
>>> read-only configuration space field at offset 0x62, size 2. This
may
>>> be harmless, but if you have problems with your device:
>>> [ 1882.349066] 1) see permissive attribute in sysfs
>>> [ 1882.349067] 2) report problems to the xen-devel mailing list
along
>>> with details of your device obtained from lspci.
>>> [ 1882.349205]   alloc irq_desc for 477 on node 0
>>> [ 1882.349215]   alloc kstat_irqs on node 0
>>> [ 1882.402893] pciback 0000:07:00.2: enabling device (0000 ->
0003)
>>> [ 1882.402908] xen: registering gsi 47 triggering 0 polarity 1
>>> [ 1882.402913] xen_allocate_pirq: returning irq 47 for gsi 47
>>> [ 1882.402916] xen: --> irq=47
>>> [ 1882.402921] Already setup the GSI :47
>>> [ 1882.402925] pciback 0000:07:00.2: PCI INT C -> GSI 47 (level,
low) -> IRQ 47
>>> [ 1882.402938] pciback 0000:07:00.2: setting latency timer to 64
>>> [ 1882.403280] pciback 0000:07:00.2: Driver tried to write to a
>>> read-only configuration space field at offset 0x62, size 2. This
may
>>> be harmless, but if you have problems with your device:
>>> [ 1882.403282] 1) see permissive attribute in sysfs
>>> [ 1882.403282] 2) report problems to the xen-devel mailing list
along
>>> with details of your device obtained from lspci.
>>> [ 1882.403380]   alloc irq_desc for 476 on node 0
>>> [ 1882.403386]   alloc kstat_irqs on node 0
>>> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
>>> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
>>> fault addr e6f80000, iommu reg = ffff82c3fff57000
>>> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
>>> (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn
= e6f80
>>> (XEN)     root_entry = ffff83019ff70000
>>> (XEN)     root_entry[7] = 19cf52001
>>> (XEN)     context = ffff83019cf52000
>>> (XEN)     context[0] = 102_706dc005
>>> (XEN)     l4 = ffff8300706dc000
>>> (XEN)     l4_index = 0
>>> (XEN)     l4[0] = 706db003
>>> (XEN)     l3 = ffff8300706db000
>>> (XEN)     l3_index = 3
>>> (XEN)     l3[3] = 702b6003
>>> (XEN)     l2 = ffff8300702b6000
>>> (XEN)     l2_index = 137
>>> (XEN)     l2[137] = 0
>>> (XEN)     l2[137] not present
>>> (XEN) traps.c:466:d0 Unhandled nmi fault/trap [#2] on VCPU 0
[ec=0000]
>>
>> That is not good. What changed from your earlier emails that this was
triggered?
>
>Nothing
>> Or was it triggered all along?
>
>Yes, I just included it for completeness
>
>> What happens if you run the system without the iommu enabled?
>
>Haven''t tried yet. Will check that next.
>
>-Bruce
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com<mailto:Xen-devel@lists.xensource.com>
>http://lists.xensource.com/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk

2010-Sep-28 16:19 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Tue, Sep 28, 2010 at 10:08:57AM -0600, Lin, Ray
wrote:>     I just checked the "xen dmesg". Look like DMA/iommu is the
root cause of this issue. In order to tell the source of interrupt, Tachyon chip
needs to do the DMA write to a dword memory location to indicate the source of
interrupt. What iommu option do you recommend to use ?
Lets get Jan involved in this discussion.

Jan, would some of your patches that inhibit the MSI write affect this
in a PV guest?
> 
> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0] fault
addr c00000, iommu reg = ffff82c3fff57000
> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
> (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
> (XEN)     root_entry = ffff83019ff70000
> (XEN)     root_entry[7] = 19cf52001
> (XEN)     context = ffff83019cf52000
> (XEN)     context[0] = 102_706dc005
> (XEN)     l4 = ffff8300706dc000
> (XEN)     l4_index = 0
> (XEN)     l4[0] = 706db003
> (XEN)     l3 = ffff8300706db000
> (XEN)     l3_index = 0
> (XEN)     l3[0] = 706da003
> (XEN)     l2 = ffff8300706da000
> (XEN)     l2_index = 6
> (XEN)     l2[6] = 0
> 
> 
> -Ray
> 
> 
> ________________________________
> From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Bruce Edge
> Sent: Monday, September 27, 2010 9:46 PM
> To: Jiang, Yunhong
> Cc: xen-devel@lists.xensource.com; Konrad Rzeszutek Wilk
> Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on
Nehalem
> 
> On Mon, Sep 27, 2010 at 8:26 PM, Jiang, Yunhong
<yunhong.jiang@intel.com<mailto:yunhong.jiang@intel.com>> wrote:
> "xm dmesg" should gives xen''s boot log, and sometimes it
contain some helpful information, I think, especially loglvl and guest_loglvl is
set to all.
> 
> I looked at the xm dmesg output and there''s nothing more than what
I already provided, aside from a bunch of commands from me poking at it.
> 
> -Bruce
> 
> 
> Thanks
> --jyh
> 
> From: Bruce Edge
[mailto:bruce.edge@gmail.com<mailto:bruce.edge@gmail.com>]
> Sent: Tuesday, September 28, 2010 11:16 AM
> To: Jiang, Yunhong
> Cc: Konrad Rzeszutek Wilk;
xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>
> 
> Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on
Nehalem
> 
> On Mon, Sep 27, 2010 at 6:15 PM, Jiang, Yunhong
<yunhong.jiang@intel.com<mailto:yunhong.jiang@intel.com>> wrote:
> Is the 07:0.0 your tachyon device? The VT-d fault is suspcious.
> 
> Yes, there is 1 quad port card is this sytem:
> 
> 07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> 07:00.1 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> 07:00.2 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> 07:00.3 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> 
> 
> Also is it possible to share the xen output?
> 
> I attached the dom0 boot output. Let me know if you wanted something else.
> 
> Also, here''s the dom0 console output upon starting the VM: This
lockdep error started with the release of 2.6.32.21. Note that I''m
running the same  kernel for the domU and dom0.
> 
> [ 1817.684097] ------------[ cut here ]------------
> [ 1817.684113] WARNING: at kernel/lockdep.c:2323
trace_hardirqs_on_caller+0x12f/0x190()
> [ 1817.684119] Hardware name: ProLiant DL380 G6
> [ 1817.684122] Modules linked in: xt_physdev ipv6 osa_mfgdom0 xenfs
xen_gntdev fbcon tileblit font bitblit softcursor xen_evtchn xen_pciback radeon
ttm drm_kms_helper tun drm i2c_algo_bit ipmi_si i2c_core ipmi_msghandler joydev
serio_raw hpwdt hpilo bridge stp llc usbhid hid cciss usb_storage
> [ 1817.684190] Pid: 11, comm: xenwatch Not tainted 2.6.32.21-xenoprof-1 #1
> [ 1817.684195] Call Trace:
> [ 1817.684197]  <IRQ>  [<ffffffff810aa18f>] ?
trace_hardirqs_on_caller+0x12f/0x190
> [ 1817.684209]  [<ffffffff8106bed0>] warn_slowpath_common+0x80/0xd0
> [ 1817.684217]  [<ffffffff815f2b80>] ? _spin_unlock_irq+0x30/0x40
> [ 1817.684223]  [<ffffffff8106bf34>] warn_slowpath_null+0x14/0x20
> [ 1817.684229]  [<ffffffff810aa18f>]
trace_hardirqs_on_caller+0x12f/0x190
> [ 1817.684234]  [<ffffffff810aa1fd>] trace_hardirqs_on+0xd/0x10
> [ 1817.684240]  [<ffffffff815f2b80>] _spin_unlock_irq+0x30/0x40
> [ 1817.684266]  [<ffffffff813c4fc5>]
add_to_net_schedule_list_tail+0x85/0xd0
> [ 1817.684271]  [<ffffffff813c6216>] netif_be_int+0x36/0x160
> [ 1817.684278]  [<ffffffff810e10d0>] handle_IRQ_event+0x70/0x180
> [ 1817.684284]  [<ffffffff810e36e9>] handle_edge_irq+0xc9/0x170
> [ 1817.684291]  [<ffffffff813b8d7f>]
__xen_evtchn_do_upcall+0x1bf/0x1f0
> [ 1817.684297]  [<ffffffff813b92fd>] xen_evtchn_do_upcall+0x3d/0x60
> [ 1817.684304]  [<ffffffff8101647e>]
xen_do_hypervisor_callback+0x1e/0x30
> [ 1817.684308]  <EOI>  [<ffffffff8100940a>] ?
hypercall_page+0x40a/0x1010
> [ 1817.684319]  [<ffffffff8100940a>] ? hypercall_page+0x40a/0x1010
> [ 1817.684325]  [<ffffffff813bce54>] ? xb_write+0x1e4/0x290
> [ 1817.684330]  [<ffffffff813bd8ca>] ? xs_talkv+0x6a/0x1f0
> [ 1817.684336]  [<ffffffff813bd8d8>] ? xs_talkv+0x78/0x1f0
> [ 1817.684341]  [<ffffffff813bdbcd>] ? xs_single+0x4d/0x60
> [ 1817.684346]  [<ffffffff813be502>] ? xenbus_read+0x52/0x80
> [ 1817.684352]  [<ffffffff813c87fc>] ? frontend_changed+0x48c/0x770
> [ 1817.684358]  [<ffffffff813bf76d>] ?
xenbus_otherend_changed+0xdd/0x1b0
> [ 1817.684365]  [<ffffffff8101122f>] ?
xen_restore_fl_direct_end+0x0/0x1
> [ 1817.684371]  [<ffffffff810ac830>] ? lock_release+0xb0/0x230
> [ 1817.684376]  [<ffffffff813bfae0>] ? frontend_changed+0x10/0x20
> [ 1817.684382]  [<ffffffff813bd4f5>] ? xenwatch_thread+0x55/0x160
> [ 1817.684389]  [<ffffffff81093400>] ?
autoremove_wake_function+0x0/0x40
> [ 1817.684394]  [<ffffffff813bd4a0>] ? xenwatch_thread+0x0/0x160
> [ 1817.684400]  [<ffffffff81093086>] ? kthread+0x96/0xb0
> [ 1817.684405]  [<ffffffff8101632a>] ? child_rip+0xa/0x20
> [ 1817.684410]  [<ffffffff81015c90>] ? restore_args+0x0/0x30
> [ 1817.684415]  [<ffffffff81016320>] ? child_rip+0x0/0x20
> 
> -Bruce
> 
> 
> 
> Thanks
> --jyh
> 
> >-----Original Message-----
> >From:
xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounces@lists.xensource.com>
>
>[mailto:xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounces@lists.xensource.com>]
On Behalf Of Bruce Edge
> >Sent: Tuesday, September 28, 2010 7:54 AM
> >To: Konrad Rzeszutek Wilk
> >Cc:
xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>
> >Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on
Nehalem
> >
> >On Mon, Sep 27, 2010 at 12:54 PM, Konrad Rzeszutek Wilk
> ><konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>>
wrote:
> >> On Mon, Sep 27, 2010 at 12:16:50PM -0700, Bruce Edge wrote:
> >>> On Mon, Sep 27, 2010 at 10:24 AM, Konrad Rzeszutek Wilk
> >>>
<konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>> wrote:
> >>> >
> >>> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge
wrote:
> >>> > > One of our developers who is working on a tachyon
driver is
> >>> > > complaining that the pvops domU kernel is not
working for these MSI
> >>> > > interrupts.
> >>> > > This is using the current head of xen/2.6.32.x on
both a single
> >>> > > Nahelam 920 and a dual E5540. This behavior is
consistent with Xen
> >>> > > 4.0.1, 4.0.2.rc1-pre and 4.1.
> >>> > >
> >>> > > Here are his comments:
> >>> > >
> >>> > > - the driver has no problem to enable msi interrupt
and request the
> >>> > > interrupt through kernel functions pci_enable_msi
& request_irq
> >>> >
> >>> > What shows up in the Xen console when you send the
''q'' key? Does it
> >>> > show that the vector is assigned to the appropiate guest?
> >>>
> >>> The Xen console q key shows that the domU is assigned:
> >>>
> >>> (XEN)     Interrupts { 32, 41-42, 47 }
> >>
> >> Aha!
> >>
> >>>
> >>> but the domU thinks it has:
> >>>
> >>> 124/125/126/127
> >>>
> >>> Is there some mapping that''s taking place, or is this
plain wrong?
> >>
> >> That looks wrong. The IRQ numbers (even though they are MSI
vectors) are
> >> setup as IRQ numbers in the DomU guest. You should have seen
> >>
> >> 32:
> >> 41:
> >> 42:
> >> 47:
> >> in you /proc/interrupts on your DomU guest.
> >>
> >> I wonder what broke  - can you use
>
>git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git>
> >> devel/xen-pcifront-0.5 (or pv/pcifront-2.6.32)?
> >
> >Please forgive the git ignorance.
> >
> >Is this the right syntax?
> >
> >git clone
git://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.32<http://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.32>
> >linux-2.6.32-pv-pcifront
> >
> >Initialized empty Git repository in
> >/import/kaan/bedge/src/xen/kernel/pv-ops/linux-2.6.32-pv-pcifront/.git/
> >fatal: The remote end hung up unexpectedly
> >
> >Or:
> >
> > git clone 
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git>
> >
> >Initialized empty Git repository in
> >/import/kaan/bedge/src/xen/kernel/pv-ops/xen/.git/
> >remote: error: Could not read 59eab2f8f04147c5aadc99f2034ca7e5b81e890f
> >remote: fatal: Failed to traverse parents of commit
> >979e121cb348add17ed8171bf447b27a3a9d1be3
> >remote: aborting due to possible repository corruption on the remote
side.
> >fatal: early EOF
> >fatal: index-pack failed
> >
> >>
> >> It has the latest pcifront driver but without the PVonHVM
enhancments
> >> so we can try to eliminate the PvONHVM logic out of the picture.
> >>
> >>>
> >>> >
> >>> > > - the interrupt does happen. But the interrupt
service routine of
> >>> > > tachyon driver doesn''t detect any interrupt
status related to this
> >>> > > interrupt, which inhibits the tachyon chip from
coming on-line. And
> >>> > > there are high count of tachyon interrupt in
/proc/interrupts
> >>> >
> >>> > Is it checking the PCI_STATUS_INTERRUPT or the appropiate
register
> >>> > in the MMIO BAR?
> >>> >
> >>>
> >>> The driver would check the appropriate register (tachyon
registers) in
> >>> the MMIO to determine the source of interrupts.
> >>
> >> OK, so that isn''t it. Is there anything at these vectors:
> >> 7c, 7d, 7e, and 7f? When you use xen debug-keys
''i'' or ''q'' it should give you
> >> an inkling what device this is set for.
> >
> >When I run a distro kernel in hvm mode, I get the expected irq
mappings:
> >
> >''i'' - Note 66 - 69
> >(XEN)    IRQ:  66 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:3a
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=10:127(----),
> >(XEN)    IRQ:  67 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:42
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=10:126(----),
> >(XEN)    IRQ:  68 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:4a
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=10:125(----),
> >(XEN)    IRQ:  69 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:52
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=10:124(----)
> >
> >
> >''q''
> >(XEN)     Interrupts { 32, 41-42, 47, 124-127 }
> >
> >
> >The same data with pv-ops kernel shows:
> >
> >''i''
> >IRQ numbers stop at 65, no 66 - 69 present:
> >
> >(XEN)    IRQ:  63 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:91
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=0:289(----),
> >(XEN)    IRQ:  64 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:99
> >type=PCI-MSI         status=00000002 mapped, unbound
> >(XEN)    IRQ:  65 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:b1
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=0:287(----),
> >(XEN) IO-APIC interrupt information:
> >
> >''q''
> >(XEN)     Interrupts { 32, 41-42, 47 }
> >
> >>
> >>>
> >>> > >
> >>> > > kaan-18-dpm:~# cat /proc/interrupts | grep TACH
> >>> > >
> >124:     760415          0          0          0          0
> >    0
> >>> > >          0          0          0          0         
0
> >      0
> >>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> >>> > >
> >125:     762234          0          0          0          0
> >    0
> >>> > >          0          0          0          0         
0
> >      0
> >>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> >>> > >
> >126:     764180          0          0          0          0
> >    0
> >>> > >          0          0          0          0         
0
> >      0
> >>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> >>> > >
> >127:     764164          0          0          0          0
> >    0
> >>> > >          0          0          0          0         
0
> >      0
> >>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> >>> >
> >>> > Can you provide the full dmesg output?
> >>>
> >>> Attached.
> >>>
> >>> Some possibly related messages on dom0 console:
> >>>
> >>> [ 1882.269778] pciback 0000:07:00.0: enabling device (0000
-> 0003)
> >>> [ 1882.269800] xen: registering gsi 32 triggering 0 polarity 1
> >>> [ 1882.269827] xen_allocate_pirq: returning irq 32 for gsi 32
> >>> [ 1882.269834] xen: --> irq=32
> >>> [ 1882.269841] Already setup the GSI :32
> >>> [ 1882.269847] pciback 0000:07:00.0: PCI INT A -> GSI 32
(level, low) -> IRQ 32
> >>> [ 1882.269866] pciback 0000:07:00.0: setting latency timer to
64
> >>> [ 1882.270463] pciback 0000:07:00.0: Driver tried to write to
a
> >>> read-only configuration space field at offset 0x62, size 2.
This may
> >>> be harmless, but if you have problems with your device:
> >>
> >> Uhhh, for that I think you need to do ''lspci -vvv -xxx -s
07:00.00''
> >> to find out what is at the configuration space. You could enable
> >> it using the permissive attribute.
> >>
> >>> [ 1882.270465] 1) see permissive attribute in sysfs
> >>> [ 1882.270467] 2) report problems to the xen-devel mailing
list along
> >>> with details of your device obtained from lspci.
> >>> [ 1882.270615]   alloc irq_desc for 478 on node 0
> >>> [ 1882.270625]   alloc kstat_irqs on node 0
> >>
> >> So for 478: what do you see? xen-pciback I presume?
> >>> [ 1882.348411] pciback 0000:07:00.1: enabling device (0000
-> 0003)
> >>> [ 1882.348433] xen: registering gsi 42 triggering 0 polarity 1
> >>> [ 1882.348440] xen_allocate_pirq: returning irq 42 for gsi 42
> >>> [ 1882.348445] xen: --> irq=42
> >>> [ 1882.348472] Already setup the GSI :42
> >>> [ 1882.348479] pciback 0000:07:00.1: PCI INT B -> GSI 42
(level, low) -> IRQ 42
> >>> [ 1882.348497] pciback 0000:07:00.1: setting latency timer to
64
> >>> [ 1882.349063] pciback 0000:07:00.1: Driver tried to write to
a
> >>> read-only configuration space field at offset 0x62, size 2.
This may
> >>> be harmless, but if you have problems with your device:
> >>> [ 1882.349066] 1) see permissive attribute in sysfs
> >>> [ 1882.349067] 2) report problems to the xen-devel mailing
list along
> >>> with details of your device obtained from lspci.
> >>> [ 1882.349205]   alloc irq_desc for 477 on node 0
> >>> [ 1882.349215]   alloc kstat_irqs on node 0
> >>> [ 1882.402893] pciback 0000:07:00.2: enabling device (0000
-> 0003)
> >>> [ 1882.402908] xen: registering gsi 47 triggering 0 polarity 1
> >>> [ 1882.402913] xen_allocate_pirq: returning irq 47 for gsi 47
> >>> [ 1882.402916] xen: --> irq=47
> >>> [ 1882.402921] Already setup the GSI :47
> >>> [ 1882.402925] pciback 0000:07:00.2: PCI INT C -> GSI 47
(level, low) -> IRQ 47
> >>> [ 1882.402938] pciback 0000:07:00.2: setting latency timer to
64
> >>> [ 1882.403280] pciback 0000:07:00.2: Driver tried to write to
a
> >>> read-only configuration space field at offset 0x62, size 2.
This may
> >>> be harmless, but if you have problems with your device:
> >>> [ 1882.403282] 1) see permissive attribute in sysfs
> >>> [ 1882.403282] 2) report problems to the xen-devel mailing
list along
> >>> with details of your device obtained from lspci.
> >>> [ 1882.403380]   alloc irq_desc for 476 on node 0
> >>> [ 1882.403386]   alloc kstat_irqs on node 0
> >>> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending
Fault
> >>> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device
[07:00.0]
> >>> fault addr e6f80000, iommu reg = ffff82c3fff57000
> >>> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
> >>> (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0
gmfn = e6f80
> >>> (XEN)     root_entry = ffff83019ff70000
> >>> (XEN)     root_entry[7] = 19cf52001
> >>> (XEN)     context = ffff83019cf52000
> >>> (XEN)     context[0] = 102_706dc005
> >>> (XEN)     l4 = ffff8300706dc000
> >>> (XEN)     l4_index = 0
> >>> (XEN)     l4[0] = 706db003
> >>> (XEN)     l3 = ffff8300706db000
> >>> (XEN)     l3_index = 3
> >>> (XEN)     l3[3] = 702b6003
> >>> (XEN)     l2 = ffff8300702b6000
> >>> (XEN)     l2_index = 137
> >>> (XEN)     l2[137] = 0
> >>> (XEN)     l2[137] not present
> >>> (XEN) traps.c:466:d0 Unhandled nmi fault/trap [#2] on VCPU 0
[ec=0000]
> >>
> >> That is not good. What changed from your earlier emails that this
was triggered?
> >
> >Nothing
> >> Or was it triggered all along?
> >
> >Yes, I just included it for completeness
> >
> >> What happens if you run the system without the iommu enabled?
> >
> >Haven''t tried yet. Will check that next.
> >
> >-Bruce
> >
> >_______________________________________________
> >Xen-devel mailing list
>
>Xen-devel@lists.xensource.com<mailto:Xen-devel@lists.xensource.com>
> >http://lists.xensource.com/xen-devel
> 
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Lin, Ray

2010-Sep-28 18:35 UTC

head link

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
> fault addr c00000, iommu reg = ffff82c3fff57000
The driver gets the physical addr 0xc0049c thru kernel function virt_to_phys()
and set the dma address of Tachyon chip with this address. This address
translation is also involved with SWIOTLB library. Is there any issue related
with SWIOTLB in pvops kernel ?




-Ray




-----Original Message-----
From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
Sent: Tuesday, September 28, 2010 9:19 AM
To: Lin, Ray; JBeulich@novell.com
Cc: Bruce Edge; Jiang, Yunhong; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Tue, Sep 28, 2010 at 10:08:57AM -0600, Lin, Ray
wrote:>     I just checked the "xen dmesg". Look like DMA/iommu is the
root cause of this issue. In order to tell the source of interrupt, Tachyon chip
needs to do the DMA write to a dword memory location to indicate the source of
interrupt. What iommu option do you recommend to use ?
Lets get Jan involved in this discussion.

Jan, would some of your patches that inhibit the MSI write affect this in a PV
guest?
>
> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
> fault addr c00000, iommu reg = ffff82c3fff57000
> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
> (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn = c00
> (XEN)     root_entry = ffff83019ff70000
> (XEN)     root_entry[7] = 19cf52001
> (XEN)     context = ffff83019cf52000
> (XEN)     context[0] = 102_706dc005
> (XEN)     l4 = ffff8300706dc000
> (XEN)     l4_index = 0
> (XEN)     l4[0] = 706db003
> (XEN)     l3 = ffff8300706db000
> (XEN)     l3_index = 0
> (XEN)     l3[0] = 706da003
> (XEN)     l2 = ffff8300706da000
> (XEN)     l2_index = 6
> (XEN)     l2[6] = 0
>
>
> -Ray
>
>
> ________________________________
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Bruce Edge
> Sent: Monday, September 27, 2010 9:46 PM
> To: Jiang, Yunhong
> Cc: xen-devel@lists.xensource.com; Konrad Rzeszutek Wilk
> Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts
> on Nehalem
>
> On Mon, Sep 27, 2010 at 8:26 PM, Jiang, Yunhong
<yunhong.jiang@intel.com<mailto:yunhong.jiang@intel.com>> wrote:
> "xm dmesg" should gives xen''s boot log, and sometimes it
contain some helpful information, I think, especially loglvl and guest_loglvl is
set to all.
>
> I looked at the xm dmesg output and there''s nothing more than what
I already provided, aside from a bunch of commands from me poking at it.
>
> -Bruce
>
>
> Thanks
> --jyh
>
> From: Bruce Edge
> [mailto:bruce.edge@gmail.com<mailto:bruce.edge@gmail.com>]
> Sent: Tuesday, September 28, 2010 11:16 AM
> To: Jiang, Yunhong
> Cc: Konrad Rzeszutek Wilk;
> xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>
>
> Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts
> on Nehalem
>
> On Mon, Sep 27, 2010 at 6:15 PM, Jiang, Yunhong
<yunhong.jiang@intel.com<mailto:yunhong.jiang@intel.com>> wrote:
> Is the 07:0.0 your tachyon device? The VT-d fault is suspcious.
>
> Yes, there is 1 quad port card is this sytem:
>
> 07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> 07:00.1 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> 07:00.2 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> 07:00.3 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
>
>
> Also is it possible to share the xen output?
>
> I attached the dom0 boot output. Let me know if you wanted something else.
>
> Also, here''s the dom0 console output upon starting the VM: This
lockdep error started with the release of 2.6.32.21. Note that I''m
running the same  kernel for the domU and dom0.
>
> [ 1817.684097] ------------[ cut here ]------------ [ 1817.684113]
> WARNING: at kernel/lockdep.c:2323
> trace_hardirqs_on_caller+0x12f/0x190()
> [ 1817.684119] Hardware name: ProLiant DL380 G6 [ 1817.684122] Modules
> linked in: xt_physdev ipv6 osa_mfgdom0 xenfs xen_gntdev fbcon tileblit
> font bitblit softcursor xen_evtchn xen_pciback radeon ttm
> drm_kms_helper tun drm i2c_algo_bit ipmi_si i2c_core ipmi_msghandler joydev
serio_raw hpwdt hpilo bridge stp llc usbhid hid cciss usb_storage [ 1817.684190]
Pid: 11, comm: xenwatch Not tainted 2.6.32.21-xenoprof-1 #1 [ 1817.684195] Call
Trace:
> [ 1817.684197]  <IRQ>  [<ffffffff810aa18f>] ?
> trace_hardirqs_on_caller+0x12f/0x190
> [ 1817.684209]  [<ffffffff8106bed0>] warn_slowpath_common+0x80/0xd0 [
> 1817.684217]  [<ffffffff815f2b80>] ? _spin_unlock_irq+0x30/0x40 [
> 1817.684223]  [<ffffffff8106bf34>] warn_slowpath_null+0x14/0x20 [
> 1817.684229]  [<ffffffff810aa18f>]
> trace_hardirqs_on_caller+0x12f/0x190
> [ 1817.684234]  [<ffffffff810aa1fd>] trace_hardirqs_on+0xd/0x10 [
> 1817.684240]  [<ffffffff815f2b80>] _spin_unlock_irq+0x30/0x40 [
> 1817.684266]  [<ffffffff813c4fc5>]
> add_to_net_schedule_list_tail+0x85/0xd0
> [ 1817.684271]  [<ffffffff813c6216>] netif_be_int+0x36/0x160 [
> 1817.684278]  [<ffffffff810e10d0>] handle_IRQ_event+0x70/0x180 [
> 1817.684284]  [<ffffffff810e36e9>] handle_edge_irq+0xc9/0x170 [
> 1817.684291]  [<ffffffff813b8d7f>] __xen_evtchn_do_upcall+0x1bf/0x1f0
> [ 1817.684297]  [<ffffffff813b92fd>] xen_evtchn_do_upcall+0x3d/0x60 [
> 1817.684304]  [<ffffffff8101647e>]
> xen_do_hypervisor_callback+0x1e/0x30
> [ 1817.684308]  <EOI>  [<ffffffff8100940a>] ?
> hypercall_page+0x40a/0x1010 [ 1817.684319]  [<ffffffff8100940a>] ?
> hypercall_page+0x40a/0x1010 [ 1817.684325]  [<ffffffff813bce54>] ?
> xb_write+0x1e4/0x290 [ 1817.684330]  [<ffffffff813bd8ca>] ?
> xs_talkv+0x6a/0x1f0 [ 1817.684336]  [<ffffffff813bd8d8>] ?
> xs_talkv+0x78/0x1f0 [ 1817.684341]  [<ffffffff813bdbcd>] ?
> xs_single+0x4d/0x60 [ 1817.684346]  [<ffffffff813be502>] ?
> xenbus_read+0x52/0x80 [ 1817.684352]  [<ffffffff813c87fc>] ?
> frontend_changed+0x48c/0x770 [ 1817.684358]  [<ffffffff813bf76d>] ?
> xenbus_otherend_changed+0xdd/0x1b0
> [ 1817.684365]  [<ffffffff8101122f>] ?
> xen_restore_fl_direct_end+0x0/0x1 [ 1817.684371] 
[<ffffffff810ac830>]
> ? lock_release+0xb0/0x230 [ 1817.684376]  [<ffffffff813bfae0>] ?
> frontend_changed+0x10/0x20 [ 1817.684382]  [<ffffffff813bd4f5>] ?
> xenwatch_thread+0x55/0x160 [ 1817.684389]  [<ffffffff81093400>] ?
> autoremove_wake_function+0x0/0x40 [ 1817.684394] 
[<ffffffff813bd4a0>]
> ? xenwatch_thread+0x0/0x160 [ 1817.684400]  [<ffffffff81093086>] ?
> kthread+0x96/0xb0 [ 1817.684405]  [<ffffffff8101632a>] ?
> child_rip+0xa/0x20 [ 1817.684410]  [<ffffffff81015c90>] ?
> restore_args+0x0/0x30 [ 1817.684415]  [<ffffffff81016320>] ?
> child_rip+0x0/0x20
>
> -Bruce
>
>
>
> Thanks
> --jyh
>
> >-----Original Message-----
> >From:
>
>xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounces@lists.
> >xensource.com>
>
>[mailto:xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounce
> >s@lists.xensource.com>] On Behalf Of Bruce Edge
> >Sent: Tuesday, September 28, 2010 7:54 AM
> >To: Konrad Rzeszutek Wilk
> >Cc:
>
>xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>
> >Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts
> >on Nehalem
> >
> >On Mon, Sep 27, 2010 at 12:54 PM, Konrad Rzeszutek Wilk
> ><konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>>
wrote:
> >> On Mon, Sep 27, 2010 at 12:16:50PM -0700, Bruce Edge wrote:
> >>> On Mon, Sep 27, 2010 at 10:24 AM, Konrad Rzeszutek Wilk
> >>>
<konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>> wrote:
> >>> >
> >>> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge
wrote:
> >>> > > One of our developers who is working on a tachyon
driver is
> >>> > > complaining that the pvops domU kernel is not
working for
> >>> > > these MSI interrupts.
> >>> > > This is using the current head of xen/2.6.32.x on
both a
> >>> > > single Nahelam 920 and a dual E5540. This behavior
is
> >>> > > consistent with Xen 4.0.1, 4.0.2.rc1-pre and 4.1.
> >>> > >
> >>> > > Here are his comments:
> >>> > >
> >>> > > - the driver has no problem to enable msi interrupt
and
> >>> > > request the interrupt through kernel functions
pci_enable_msi
> >>> > > & request_irq
> >>> >
> >>> > What shows up in the Xen console when you send the
''q'' key? Does
> >>> > it show that the vector is assigned to the appropiate
guest?
> >>>
> >>> The Xen console q key shows that the domU is assigned:
> >>>
> >>> (XEN)     Interrupts { 32, 41-42, 47 }
> >>
> >> Aha!
> >>
> >>>
> >>> but the domU thinks it has:
> >>>
> >>> 124/125/126/127
> >>>
> >>> Is there some mapping that''s taking place, or is this
plain wrong?
> >>
> >> That looks wrong. The IRQ numbers (even though they are MSI
> >> vectors) are setup as IRQ numbers in the DomU guest. You should
> >> have seen
> >>
> >> 32:
> >> 41:
> >> 42:
> >> 47:
> >> in you /proc/interrupts on your DomU guest.
> >>
> >> I wonder what broke  - can you use
>
>git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http://g
> >it.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git>
> >> devel/xen-pcifront-0.5 (or pv/pcifront-2.6.32)?
> >
> >Please forgive the git ignorance.
> >
> >Is this the right syntax?
> >
> >git clone
> >git://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.
>
>32<http://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-
> >2.6.32>
> >linux-2.6.32-pv-pcifront
> >
> >Initialized empty Git repository in
> >/import/kaan/bedge/src/xen/kernel/pv-ops/linux-2.6.32-pv-pcifront/.gi
> >t/
> >fatal: The remote end hung up unexpectedly
> >
> >Or:
> >
> > git clone
> >
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http://
> > git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git>
> >
> >Initialized empty Git repository in
> >/import/kaan/bedge/src/xen/kernel/pv-ops/xen/.git/
> >remote: error: Could not read
> >59eab2f8f04147c5aadc99f2034ca7e5b81e890f
> >remote: fatal: Failed to traverse parents of commit
> >979e121cb348add17ed8171bf447b27a3a9d1be3
> >remote: aborting due to possible repository corruption on the remote
side.
> >fatal: early EOF
> >fatal: index-pack failed
> >
> >>
> >> It has the latest pcifront driver but without the PVonHVM
> >> enhancments so we can try to eliminate the PvONHVM logic out of
the picture.
> >>
> >>>
> >>> >
> >>> > > - the interrupt does happen. But the interrupt
service routine
> >>> > > of tachyon driver doesn''t detect any
interrupt status related
> >>> > > to this interrupt, which inhibits the tachyon chip
from coming
> >>> > > on-line. And there are high count of tachyon
interrupt in
> >>> > > /proc/interrupts
> >>> >
> >>> > Is it checking the PCI_STATUS_INTERRUPT or the appropiate
> >>> > register in the MMIO BAR?
> >>> >
> >>>
> >>> The driver would check the appropriate register (tachyon
> >>> registers) in the MMIO to determine the source of interrupts.
> >>
> >> OK, so that isn''t it. Is there anything at these vectors:
> >> 7c, 7d, 7e, and 7f? When you use xen debug-keys
''i'' or ''q'' it
> >> should give you an inkling what device this is set for.
> >
> >When I run a distro kernel in hvm mode, I get the expected irq
mappings:
> >
> >''i'' - Note 66 - 69
> >(XEN)    IRQ:  66 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:3a
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=10:127(----),
> >(XEN)    IRQ:  67 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:42
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=10:126(----),
> >(XEN)    IRQ:  68 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:4a
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=10:125(----),
> >(XEN)    IRQ:  69 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:52
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=10:124(----)
> >
> >
> >''q''
> >(XEN)     Interrupts { 32, 41-42, 47, 124-127 }
> >
> >
> >The same data with pv-ops kernel shows:
> >
> >''i''
> >IRQ numbers stop at 65, no 66 - 69 present:
> >
> >(XEN)    IRQ:  63 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:91
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=0:289(----),
> >(XEN)    IRQ:  64 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:99
> >type=PCI-MSI         status=00000002 mapped, unbound
> >(XEN)    IRQ:  65 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:b1
> >type=PCI-MSI         status=00000010 in-flight=0
> >domain-list=0:287(----),
> >(XEN) IO-APIC interrupt information:
> >
> >''q''
> >(XEN)     Interrupts { 32, 41-42, 47 }
> >
> >>
> >>>
> >>> > >
> >>> > > kaan-18-dpm:~# cat /proc/interrupts | grep TACH
> >>> > >
> >124:     760415          0          0          0          0
> >    0
> >>> > >          0          0          0          0         
0
> >      0
> >>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> >>> > >
> >125:     762234          0          0          0          0
> >    0
> >>> > >          0          0          0          0         
0
> >      0
> >>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> >>> > >
> >126:     764180          0          0          0          0
> >    0
> >>> > >          0          0          0          0         
0
> >      0
> >>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> >>> > >
> >127:     764164          0          0          0          0
> >    0
> >>> > >          0          0          0          0         
0
> >      0
> >>> > >     0          0  xen-pirq-pcifront-msi  HW_TACHYON
> >>> >
> >>> > Can you provide the full dmesg output?
> >>>
> >>> Attached.
> >>>
> >>> Some possibly related messages on dom0 console:
> >>>
> >>> [ 1882.269778] pciback 0000:07:00.0: enabling device (0000
->
> >>> 0003) [ 1882.269800] xen: registering gsi 32 triggering 0
polarity
> >>> 1 [ 1882.269827] xen_allocate_pirq: returning irq 32 for gsi
32 [
> >>> 1882.269834] xen: --> irq=32 [ 1882.269841] Already setup
the GSI
> >>> :32 [ 1882.269847] pciback 0000:07:00.0: PCI INT A -> GSI
32
> >>> (level, low) -> IRQ 32 [ 1882.269866] pciback 0000:07:00.0:
> >>> setting latency timer to 64 [ 1882.270463] pciback
0000:07:00.0:
> >>> Driver tried to write to a read-only configuration space field
at
> >>> offset 0x62, size 2. This may be harmless, but if you have
> >>> problems with your device:
> >>
> >> Uhhh, for that I think you need to do ''lspci -vvv -xxx -s
07:00.00''
> >> to find out what is at the configuration space. You could enable
it
> >> using the permissive attribute.
> >>
> >>> [ 1882.270465] 1) see permissive attribute in sysfs [
1882.270467]
> >>> 2) report problems to the xen-devel mailing list along with
> >>> details of your device obtained from lspci.
> >>> [ 1882.270615]   alloc irq_desc for 478 on node 0
> >>> [ 1882.270625]   alloc kstat_irqs on node 0
> >>
> >> So for 478: what do you see? xen-pciback I presume?
> >>> [ 1882.348411] pciback 0000:07:00.1: enabling device (0000
->
> >>> 0003) [ 1882.348433] xen: registering gsi 42 triggering 0
polarity
> >>> 1 [ 1882.348440] xen_allocate_pirq: returning irq 42 for gsi
42 [
> >>> 1882.348445] xen: --> irq=42 [ 1882.348472] Already setup
the GSI
> >>> :42 [ 1882.348479] pciback 0000:07:00.1: PCI INT B -> GSI
42
> >>> (level, low) -> IRQ 42 [ 1882.348497] pciback 0000:07:00.1:
> >>> setting latency timer to 64 [ 1882.349063] pciback
0000:07:00.1:
> >>> Driver tried to write to a read-only configuration space field
at
> >>> offset 0x62, size 2. This may be harmless, but if you have
> >>> problems with your device:
> >>> [ 1882.349066] 1) see permissive attribute in sysfs [
1882.349067]
> >>> 2) report problems to the xen-devel mailing list along with
> >>> details of your device obtained from lspci.
> >>> [ 1882.349205]   alloc irq_desc for 477 on node 0
> >>> [ 1882.349215]   alloc kstat_irqs on node 0
> >>> [ 1882.402893] pciback 0000:07:00.2: enabling device (0000
->
> >>> 0003) [ 1882.402908] xen: registering gsi 47 triggering 0
polarity
> >>> 1 [ 1882.402913] xen_allocate_pirq: returning irq 47 for gsi
47 [
> >>> 1882.402916] xen: --> irq=47 [ 1882.402921] Already setup
the GSI
> >>> :47 [ 1882.402925] pciback 0000:07:00.2: PCI INT C -> GSI
47
> >>> (level, low) -> IRQ 47 [ 1882.402938] pciback 0000:07:00.2:
> >>> setting latency timer to 64 [ 1882.403280] pciback
0000:07:00.2:
> >>> Driver tried to write to a read-only configuration space field
at
> >>> offset 0x62, size 2. This may be harmless, but if you have
> >>> problems with your device:
> >>> [ 1882.403282] 1) see permissive attribute in sysfs [
1882.403282]
> >>> 2) report problems to the xen-devel mailing list along with
> >>> details of your device obtained from lspci.
> >>> [ 1882.403380]   alloc irq_desc for 476 on node 0
> >>> [ 1882.403386]   alloc kstat_irqs on node 0
> >>> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending
Fault
> >>> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device
[07:00.0]
> >>> fault addr e6f80000, iommu reg = ffff82c3fff57000
> >>> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
> >>> (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0
gmfn = e6f80
> >>> (XEN)     root_entry = ffff83019ff70000
> >>> (XEN)     root_entry[7] = 19cf52001
> >>> (XEN)     context = ffff83019cf52000
> >>> (XEN)     context[0] = 102_706dc005
> >>> (XEN)     l4 = ffff8300706dc000
> >>> (XEN)     l4_index = 0
> >>> (XEN)     l4[0] = 706db003
> >>> (XEN)     l3 = ffff8300706db000
> >>> (XEN)     l3_index = 3
> >>> (XEN)     l3[3] = 702b6003
> >>> (XEN)     l2 = ffff8300702b6000
> >>> (XEN)     l2_index = 137
> >>> (XEN)     l2[137] = 0
> >>> (XEN)     l2[137] not present
> >>> (XEN) traps.c:466:d0 Unhandled nmi fault/trap [#2] on VCPU 0
> >>> [ec=0000]
> >>
> >> That is not good. What changed from your earlier emails that this
was triggered?
> >
> >Nothing
> >> Or was it triggered all along?
> >
> >Yes, I just included it for completeness
> >
> >> What happens if you run the system without the iommu enabled?
> >
> >Haven''t tried yet. Will check that next.
> >
> >-Bruce
> >
> >_______________________________________________
> >Xen-devel mailing list
>
>Xen-devel@lists.xensource.com<mailto:Xen-devel@lists.xensource.com>
> >http://lists.xensource.com/xen-devel
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk

2010-Sep-28 20:14 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Tue, Sep 28, 2010 at 12:35:12PM -0600, Lin, Ray
wrote:> 
> > (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
> > (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
> > fault addr c00000, iommu reg = ffff82c3fff57000
> 
> The driver gets the physical addr 0xc0049c thru kernel function
virt_to_phys() and set the dma address of Tachyon chip with this address. This
address translation is also involved with SWIOTLB library. Is there any issue
related with SWIOTLB in pvops kernel ?
If the driver is using pci_map_page, which goes through the Xen-SWIOTLB library,
then
it works correctly (the Xen-SWIOTLB does a PFN->MFN translation to give you
the bus address).

I presume the physical address (0xc0049c) is the BAR of the PCI device, right?

There are no issues with the Xen-SWIOTLB in the pvops kernel.
> 
> 
> 
> 
> -Ray
> 
> 
> 
> 
> -----Original Message-----
> From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
> Sent: Tuesday, September 28, 2010 9:19 AM
> To: Lin, Ray; JBeulich@novell.com
> Cc: Bruce Edge; Jiang, Yunhong; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on
Nehalem
> 
> On Tue, Sep 28, 2010 at 10:08:57AM -0600, Lin, Ray wrote:
> >     I just checked the "xen dmesg". Look like DMA/iommu is
the root cause of this issue. In order to tell the source of interrupt, Tachyon
chip needs to do the DMA write to a dword memory location to indicate the source
of interrupt. What iommu option do you recommend to use ?
> 
> Lets get Jan involved in this discussion.
> 
> Jan, would some of your patches that inhibit the MSI write affect this in a
PV guest?
> 
> >
> > (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
> > (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
> > fault addr c00000, iommu reg = ffff82c3fff57000
> > (XEN) DMAR:[fault reason 05h] PTE Write access is not set
> > (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn =
c00
> > (XEN)     root_entry = ffff83019ff70000
> > (XEN)     root_entry[7] = 19cf52001
> > (XEN)     context = ffff83019cf52000
> > (XEN)     context[0] = 102_706dc005
> > (XEN)     l4 = ffff8300706dc000
> > (XEN)     l4_index = 0
> > (XEN)     l4[0] = 706db003
> > (XEN)     l3 = ffff8300706db000
> > (XEN)     l3_index = 0
> > (XEN)     l3[0] = 706da003
> > (XEN)     l2 = ffff8300706da000
> > (XEN)     l2_index = 6
> > (XEN)     l2[6] = 0
> >
> >
> > -Ray
> >
> >
> > ________________________________
> > From: xen-devel-bounces@lists.xensource.com
> > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Bruce Edge
> > Sent: Monday, September 27, 2010 9:46 PM
> > To: Jiang, Yunhong
> > Cc: xen-devel@lists.xensource.com; Konrad Rzeszutek Wilk
> > Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts
> > on Nehalem
> >
> > On Mon, Sep 27, 2010 at 8:26 PM, Jiang, Yunhong
<yunhong.jiang@intel.com<mailto:yunhong.jiang@intel.com>> wrote:
> > "xm dmesg" should gives xen''s boot log, and
sometimes it contain some helpful information, I think, especially loglvl and
guest_loglvl is set to all.
> >
> > I looked at the xm dmesg output and there''s nothing more than
what I already provided, aside from a bunch of commands from me poking at it.
> >
> > -Bruce
> >
> >
> > Thanks
> > --jyh
> >
> > From: Bruce Edge
> > [mailto:bruce.edge@gmail.com<mailto:bruce.edge@gmail.com>]
> > Sent: Tuesday, September 28, 2010 11:16 AM
> > To: Jiang, Yunhong
> > Cc: Konrad Rzeszutek Wilk;
> >
xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>
> >
> > Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts
> > on Nehalem
> >
> > On Mon, Sep 27, 2010 at 6:15 PM, Jiang, Yunhong
<yunhong.jiang@intel.com<mailto:yunhong.jiang@intel.com>> wrote:
> > Is the 07:0.0 your tachyon device? The VT-d fault is suspcious.
> >
> > Yes, there is 1 quad port card is this sytem:
> >
> > 07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> > 07:00.1 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> > 07:00.2 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> > 07:00.3 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> >
> >
> > Also is it possible to share the xen output?
> >
> > I attached the dom0 boot output. Let me know if you wanted something
else.
> >
> > Also, here''s the dom0 console output upon starting the VM:
This lockdep error started with the release of 2.6.32.21. Note that I''m
running the same  kernel for the domU and dom0.
> >
> > [ 1817.684097] ------------[ cut here ]------------ [ 1817.684113]
> > WARNING: at kernel/lockdep.c:2323
> > trace_hardirqs_on_caller+0x12f/0x190()
> > [ 1817.684119] Hardware name: ProLiant DL380 G6 [ 1817.684122] Modules
> > linked in: xt_physdev ipv6 osa_mfgdom0 xenfs xen_gntdev fbcon tileblit
> > font bitblit softcursor xen_evtchn xen_pciback radeon ttm
> > drm_kms_helper tun drm i2c_algo_bit ipmi_si i2c_core ipmi_msghandler
joydev serio_raw hpwdt hpilo bridge stp llc usbhid hid cciss usb_storage [
1817.684190] Pid: 11, comm: xenwatch Not tainted 2.6.32.21-xenoprof-1 #1 [
1817.684195] Call Trace:
> > [ 1817.684197]  <IRQ>  [<ffffffff810aa18f>] ?
> > trace_hardirqs_on_caller+0x12f/0x190
> > [ 1817.684209]  [<ffffffff8106bed0>]
warn_slowpath_common+0x80/0xd0 [
> > 1817.684217]  [<ffffffff815f2b80>] ? _spin_unlock_irq+0x30/0x40
[
> > 1817.684223]  [<ffffffff8106bf34>] warn_slowpath_null+0x14/0x20
[
> > 1817.684229]  [<ffffffff810aa18f>]
> > trace_hardirqs_on_caller+0x12f/0x190
> > [ 1817.684234]  [<ffffffff810aa1fd>] trace_hardirqs_on+0xd/0x10
[
> > 1817.684240]  [<ffffffff815f2b80>] _spin_unlock_irq+0x30/0x40 [
> > 1817.684266]  [<ffffffff813c4fc5>]
> > add_to_net_schedule_list_tail+0x85/0xd0
> > [ 1817.684271]  [<ffffffff813c6216>] netif_be_int+0x36/0x160 [
> > 1817.684278]  [<ffffffff810e10d0>] handle_IRQ_event+0x70/0x180 [
> > 1817.684284]  [<ffffffff810e36e9>] handle_edge_irq+0xc9/0x170 [
> > 1817.684291]  [<ffffffff813b8d7f>]
__xen_evtchn_do_upcall+0x1bf/0x1f0
> > [ 1817.684297]  [<ffffffff813b92fd>]
xen_evtchn_do_upcall+0x3d/0x60 [
> > 1817.684304]  [<ffffffff8101647e>]
> > xen_do_hypervisor_callback+0x1e/0x30
> > [ 1817.684308]  <EOI>  [<ffffffff8100940a>] ?
> > hypercall_page+0x40a/0x1010 [ 1817.684319]  [<ffffffff8100940a>]
?
> > hypercall_page+0x40a/0x1010 [ 1817.684325]  [<ffffffff813bce54>]
?
> > xb_write+0x1e4/0x290 [ 1817.684330]  [<ffffffff813bd8ca>] ?
> > xs_talkv+0x6a/0x1f0 [ 1817.684336]  [<ffffffff813bd8d8>] ?
> > xs_talkv+0x78/0x1f0 [ 1817.684341]  [<ffffffff813bdbcd>] ?
> > xs_single+0x4d/0x60 [ 1817.684346]  [<ffffffff813be502>] ?
> > xenbus_read+0x52/0x80 [ 1817.684352]  [<ffffffff813c87fc>] ?
> > frontend_changed+0x48c/0x770 [ 1817.684358] 
[<ffffffff813bf76d>] ?
> > xenbus_otherend_changed+0xdd/0x1b0
> > [ 1817.684365]  [<ffffffff8101122f>] ?
> > xen_restore_fl_direct_end+0x0/0x1 [ 1817.684371] 
[<ffffffff810ac830>]
> > ? lock_release+0xb0/0x230 [ 1817.684376]  [<ffffffff813bfae0>] ?
> > frontend_changed+0x10/0x20 [ 1817.684382]  [<ffffffff813bd4f5>]
?
> > xenwatch_thread+0x55/0x160 [ 1817.684389]  [<ffffffff81093400>]
?
> > autoremove_wake_function+0x0/0x40 [ 1817.684394] 
[<ffffffff813bd4a0>]
> > ? xenwatch_thread+0x0/0x160 [ 1817.684400]  [<ffffffff81093086>]
?
> > kthread+0x96/0xb0 [ 1817.684405]  [<ffffffff8101632a>] ?
> > child_rip+0xa/0x20 [ 1817.684410]  [<ffffffff81015c90>] ?
> > restore_args+0x0/0x30 [ 1817.684415]  [<ffffffff81016320>] ?
> > child_rip+0x0/0x20
> >
> > -Bruce
> >
> >
> >
> > Thanks
> > --jyh
> >
> > >-----Original Message-----
> > >From:
> >
>xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounces@lists.
> > >xensource.com>
> >
>[mailto:xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounce
> > >s@lists.xensource.com>] On Behalf Of Bruce Edge
> > >Sent: Tuesday, September 28, 2010 7:54 AM
> > >To: Konrad Rzeszutek Wilk
> > >Cc:
> >
>xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>
> > >Subject: Re: [Xen-devel] pv-ops domU not working with MSI
interrupts
> > >on Nehalem
> > >
> > >On Mon, Sep 27, 2010 at 12:54 PM, Konrad Rzeszutek Wilk
> >
><konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>> wrote:
> > >> On Mon, Sep 27, 2010 at 12:16:50PM -0700, Bruce Edge wrote:
> > >>> On Mon, Sep 27, 2010 at 10:24 AM, Konrad Rzeszutek Wilk
> > >>>
<konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>> wrote:
> > >>> >
> > >>> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge
wrote:
> > >>> > > One of our developers who is working on a
tachyon driver is
> > >>> > > complaining that the pvops domU kernel is not
working for
> > >>> > > these MSI interrupts.
> > >>> > > This is using the current head of xen/2.6.32.x
on both a
> > >>> > > single Nahelam 920 and a dual E5540. This
behavior is
> > >>> > > consistent with Xen 4.0.1, 4.0.2.rc1-pre and
4.1.
> > >>> > >
> > >>> > > Here are his comments:
> > >>> > >
> > >>> > > - the driver has no problem to enable msi
interrupt and
> > >>> > > request the interrupt through kernel functions
pci_enable_msi
> > >>> > > & request_irq
> > >>> >
> > >>> > What shows up in the Xen console when you send the
''q'' key? Does
> > >>> > it show that the vector is assigned to the
appropiate guest?
> > >>>
> > >>> The Xen console q key shows that the domU is assigned:
> > >>>
> > >>> (XEN)     Interrupts { 32, 41-42, 47 }
> > >>
> > >> Aha!
> > >>
> > >>>
> > >>> but the domU thinks it has:
> > >>>
> > >>> 124/125/126/127
> > >>>
> > >>> Is there some mapping that''s taking place, or is
this plain wrong?
> > >>
> > >> That looks wrong. The IRQ numbers (even though they are MSI
> > >> vectors) are setup as IRQ numbers in the DomU guest. You
should
> > >> have seen
> > >>
> > >> 32:
> > >> 41:
> > >> 42:
> > >> 47:
> > >> in you /proc/interrupts on your DomU guest.
> > >>
> > >> I wonder what broke  - can you use
> >
>git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http://g
> > >it.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git>
> > >> devel/xen-pcifront-0.5 (or pv/pcifront-2.6.32)?
> > >
> > >Please forgive the git ignorance.
> > >
> > >Is this the right syntax?
> > >
> > >git clone
> >
>git://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.
> >
>32<http://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-
> > >2.6.32>
> > >linux-2.6.32-pv-pcifront
> > >
> > >Initialized empty Git repository in
> >
>/import/kaan/bedge/src/xen/kernel/pv-ops/linux-2.6.32-pv-pcifront/.gi
> > >t/
> > >fatal: The remote end hung up unexpectedly
> > >
> > >Or:
> > >
> > > git clone
> > >
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http://
> > > git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git>
> > >
> > >Initialized empty Git repository in
> > >/import/kaan/bedge/src/xen/kernel/pv-ops/xen/.git/
> > >remote: error: Could not read
> > >59eab2f8f04147c5aadc99f2034ca7e5b81e890f
> > >remote: fatal: Failed to traverse parents of commit
> > >979e121cb348add17ed8171bf447b27a3a9d1be3
> > >remote: aborting due to possible repository corruption on the
remote side.
> > >fatal: early EOF
> > >fatal: index-pack failed
> > >
> > >>
> > >> It has the latest pcifront driver but without the PVonHVM
> > >> enhancments so we can try to eliminate the PvONHVM logic out
of the picture.
> > >>
> > >>>
> > >>> >
> > >>> > > - the interrupt does happen. But the interrupt
service routine
> > >>> > > of tachyon driver doesn''t detect any
interrupt status related
> > >>> > > to this interrupt, which inhibits the tachyon
chip from coming
> > >>> > > on-line. And there are high count of tachyon
interrupt in
> > >>> > > /proc/interrupts
> > >>> >
> > >>> > Is it checking the PCI_STATUS_INTERRUPT or the
appropiate
> > >>> > register in the MMIO BAR?
> > >>> >
> > >>>
> > >>> The driver would check the appropriate register (tachyon
> > >>> registers) in the MMIO to determine the source of
interrupts.
> > >>
> > >> OK, so that isn''t it. Is there anything at these
vectors:
> > >> 7c, 7d, 7e, and 7f? When you use xen debug-keys
''i'' or ''q'' it
> > >> should give you an inkling what device this is set for.
> > >
> > >When I run a distro kernel in hvm mode, I get the expected irq
mappings:
> > >
> > >''i'' - Note 66 - 69
> > >(XEN)    IRQ:  66 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:3a
> > >type=PCI-MSI         status=00000010 in-flight=0
> > >domain-list=10:127(----),
> > >(XEN)    IRQ:  67 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:42
> > >type=PCI-MSI         status=00000010 in-flight=0
> > >domain-list=10:126(----),
> > >(XEN)    IRQ:  68 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:4a
> > >type=PCI-MSI         status=00000010 in-flight=0
> > >domain-list=10:125(----),
> > >(XEN)    IRQ:  69 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:52
> > >type=PCI-MSI         status=00000010 in-flight=0
> > >domain-list=10:124(----)
> > >
> > >
> > >''q''
> > >(XEN)     Interrupts { 32, 41-42, 47, 124-127 }
> > >
> > >
> > >The same data with pv-ops kernel shows:
> > >
> > >''i''
> > >IRQ numbers stop at 65, no 66 - 69 present:
> > >
> > >(XEN)    IRQ:  63 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:91
> > >type=PCI-MSI         status=00000010 in-flight=0
> > >domain-list=0:289(----),
> > >(XEN)    IRQ:  64 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:99
> > >type=PCI-MSI         status=00000002 mapped, unbound
> > >(XEN)    IRQ:  65 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:b1
> > >type=PCI-MSI         status=00000010 in-flight=0
> > >domain-list=0:287(----),
> > >(XEN) IO-APIC interrupt information:
> > >
> > >''q''
> > >(XEN)     Interrupts { 32, 41-42, 47 }
> > >
> > >>
> > >>>
> > >>> > >
> > >>> > > kaan-18-dpm:~# cat /proc/interrupts | grep TACH
> > >>> > >
> > >124:     760415          0          0          0          0
> > >    0
> > >>> > >          0          0          0          0    
0
> > >      0
> > >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
> > >>> > >
> > >125:     762234          0          0          0          0
> > >    0
> > >>> > >          0          0          0          0    
0
> > >      0
> > >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
> > >>> > >
> > >126:     764180          0          0          0          0
> > >    0
> > >>> > >          0          0          0          0    
0
> > >      0
> > >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
> > >>> > >
> > >127:     764164          0          0          0          0
> > >    0
> > >>> > >          0          0          0          0    
0
> > >      0
> > >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
> > >>> >
> > >>> > Can you provide the full dmesg output?
> > >>>
> > >>> Attached.
> > >>>
> > >>> Some possibly related messages on dom0 console:
> > >>>
> > >>> [ 1882.269778] pciback 0000:07:00.0: enabling device
(0000 ->
> > >>> 0003) [ 1882.269800] xen: registering gsi 32 triggering 0
polarity
> > >>> 1 [ 1882.269827] xen_allocate_pirq: returning irq 32 for
gsi 32 [
> > >>> 1882.269834] xen: --> irq=32 [ 1882.269841] Already
setup the GSI
> > >>> :32 [ 1882.269847] pciback 0000:07:00.0: PCI INT A ->
GSI 32
> > >>> (level, low) -> IRQ 32 [ 1882.269866] pciback
0000:07:00.0:
> > >>> setting latency timer to 64 [ 1882.270463] pciback
0000:07:00.0:
> > >>> Driver tried to write to a read-only configuration space
field at
> > >>> offset 0x62, size 2. This may be harmless, but if you
have
> > >>> problems with your device:
> > >>
> > >> Uhhh, for that I think you need to do ''lspci -vvv
-xxx -s 07:00.00''
> > >> to find out what is at the configuration space. You could
enable it
> > >> using the permissive attribute.
> > >>
> > >>> [ 1882.270465] 1) see permissive attribute in sysfs [
1882.270467]
> > >>> 2) report problems to the xen-devel mailing list along
with
> > >>> details of your device obtained from lspci.
> > >>> [ 1882.270615]   alloc irq_desc for 478 on node 0
> > >>> [ 1882.270625]   alloc kstat_irqs on node 0
> > >>
> > >> So for 478: what do you see? xen-pciback I presume?
> > >>> [ 1882.348411] pciback 0000:07:00.1: enabling device
(0000 ->
> > >>> 0003) [ 1882.348433] xen: registering gsi 42 triggering 0
polarity
> > >>> 1 [ 1882.348440] xen_allocate_pirq: returning irq 42 for
gsi 42 [
> > >>> 1882.348445] xen: --> irq=42 [ 1882.348472] Already
setup the GSI
> > >>> :42 [ 1882.348479] pciback 0000:07:00.1: PCI INT B ->
GSI 42
> > >>> (level, low) -> IRQ 42 [ 1882.348497] pciback
0000:07:00.1:
> > >>> setting latency timer to 64 [ 1882.349063] pciback
0000:07:00.1:
> > >>> Driver tried to write to a read-only configuration space
field at
> > >>> offset 0x62, size 2. This may be harmless, but if you
have
> > >>> problems with your device:
> > >>> [ 1882.349066] 1) see permissive attribute in sysfs [
1882.349067]
> > >>> 2) report problems to the xen-devel mailing list along
with
> > >>> details of your device obtained from lspci.
> > >>> [ 1882.349205]   alloc irq_desc for 477 on node 0
> > >>> [ 1882.349215]   alloc kstat_irqs on node 0
> > >>> [ 1882.402893] pciback 0000:07:00.2: enabling device
(0000 ->
> > >>> 0003) [ 1882.402908] xen: registering gsi 47 triggering 0
polarity
> > >>> 1 [ 1882.402913] xen_allocate_pirq: returning irq 47 for
gsi 47 [
> > >>> 1882.402916] xen: --> irq=47 [ 1882.402921] Already
setup the GSI
> > >>> :47 [ 1882.402925] pciback 0000:07:00.2: PCI INT C ->
GSI 47
> > >>> (level, low) -> IRQ 47 [ 1882.402938] pciback
0000:07:00.2:
> > >>> setting latency timer to 64 [ 1882.403280] pciback
0000:07:00.2:
> > >>> Driver tried to write to a read-only configuration space
field at
> > >>> offset 0x62, size 2. This may be harmless, but if you
have
> > >>> problems with your device:
> > >>> [ 1882.403282] 1) see permissive attribute in sysfs [
1882.403282]
> > >>> 2) report problems to the xen-devel mailing list along
with
> > >>> details of your device obtained from lspci.
> > >>> [ 1882.403380]   alloc irq_desc for 476 on node 0
> > >>> [ 1882.403386]   alloc kstat_irqs on node 0
> > >>> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary
Pending Fault
> > >>> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device
[07:00.0]
> > >>> fault addr e6f80000, iommu reg = ffff82c3fff57000
> > >>> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
> > >>> (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf =
7:0.0 gmfn = e6f80
> > >>> (XEN)     root_entry = ffff83019ff70000
> > >>> (XEN)     root_entry[7] = 19cf52001
> > >>> (XEN)     context = ffff83019cf52000
> > >>> (XEN)     context[0] = 102_706dc005
> > >>> (XEN)     l4 = ffff8300706dc000
> > >>> (XEN)     l4_index = 0
> > >>> (XEN)     l4[0] = 706db003
> > >>> (XEN)     l3 = ffff8300706db000
> > >>> (XEN)     l3_index = 3
> > >>> (XEN)     l3[3] = 702b6003
> > >>> (XEN)     l2 = ffff8300702b6000
> > >>> (XEN)     l2_index = 137
> > >>> (XEN)     l2[137] = 0
> > >>> (XEN)     l2[137] not present
> > >>> (XEN) traps.c:466:d0 Unhandled nmi fault/trap [#2] on
VCPU 0
> > >>> [ec=0000]
> > >>
> > >> That is not good. What changed from your earlier emails that
this was triggered?
> > >
> > >Nothing
> > >> Or was it triggered all along?
> > >
> > >Yes, I just included it for completeness
> > >
> > >> What happens if you run the system without the iommu enabled?
> > >
> > >Haven''t tried yet. Will check that next.
> > >
> > >-Bruce
> > >
> > >_______________________________________________
> > >Xen-devel mailing list
> >
>Xen-devel@lists.xensource.com<mailto:Xen-devel@lists.xensource.com>
> > >http://lists.xensource.com/xen-devel
> >
> >
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Lin, Ray

2010-Sep-28 20:38 UTC

head link

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

The physical address (0xc0049c) is not the BAR of the PCI device. It resides in
main memory. Is there any restriction to do the DMA Write to main memory in
pvops kernel ?


-Ray


-----Original Message-----
From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
Sent: Tuesday, September 28, 2010 1:15 PM
To: Lin, Ray
Cc: JBeulich@novell.com; xen-devel@lists.xensource.com; Jiang, Yunhong; Bruce
Edge
Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Tue, Sep 28, 2010 at 12:35:12PM -0600, Lin, Ray
wrote:>
> > (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
> > (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
> > fault addr c00000, iommu reg = ffff82c3fff57000
>
> The driver gets the physical addr 0xc0049c thru kernel function
virt_to_phys() and set the dma address of Tachyon chip with this address. This
address translation is also involved with SWIOTLB library. Is there any issue
related with SWIOTLB in pvops kernel ?
If the driver is using pci_map_page, which goes through the Xen-SWIOTLB library,
then it works correctly (the Xen-SWIOTLB does a PFN->MFN translation to give
you the bus address).

I presume the physical address (0xc0049c) is the BAR of the PCI device, right?

There are no issues with the Xen-SWIOTLB in the pvops kernel.
>
>
>
>
> -Ray
>
>
>
>
> -----Original Message-----
> From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
> Sent: Tuesday, September 28, 2010 9:19 AM
> To: Lin, Ray; JBeulich@novell.com
> Cc: Bruce Edge; Jiang, Yunhong; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts
> on Nehalem
>
> On Tue, Sep 28, 2010 at 10:08:57AM -0600, Lin, Ray wrote:
> >     I just checked the "xen dmesg". Look like DMA/iommu is
the root cause of this issue. In order to tell the source of interrupt, Tachyon
chip needs to do the DMA write to a dword memory location to indicate the source
of interrupt. What iommu option do you recommend to use ?
>
> Lets get Jan involved in this discussion.
>
> Jan, would some of your patches that inhibit the MSI write affect this in a
PV guest?
>
> >
> > (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
> > (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
> > fault addr c00000, iommu reg = ffff82c3fff57000
> > (XEN) DMAR:[fault reason 05h] PTE Write access is not set
> > (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn =
c00
> > (XEN)     root_entry = ffff83019ff70000
> > (XEN)     root_entry[7] = 19cf52001
> > (XEN)     context = ffff83019cf52000
> > (XEN)     context[0] = 102_706dc005
> > (XEN)     l4 = ffff8300706dc000
> > (XEN)     l4_index = 0
> > (XEN)     l4[0] = 706db003
> > (XEN)     l3 = ffff8300706db000
> > (XEN)     l3_index = 0
> > (XEN)     l3[0] = 706da003
> > (XEN)     l2 = ffff8300706da000
> > (XEN)     l2_index = 6
> > (XEN)     l2[6] = 0
> >
> >
> > -Ray
> >
> >
> > ________________________________
> > From: xen-devel-bounces@lists.xensource.com
> > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Bruce
> > Edge
> > Sent: Monday, September 27, 2010 9:46 PM
> > To: Jiang, Yunhong
> > Cc: xen-devel@lists.xensource.com; Konrad Rzeszutek Wilk
> > Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts
> > on Nehalem
> >
> > On Mon, Sep 27, 2010 at 8:26 PM, Jiang, Yunhong
<yunhong.jiang@intel.com<mailto:yunhong.jiang@intel.com>> wrote:
> > "xm dmesg" should gives xen''s boot log, and
sometimes it contain some helpful information, I think, especially loglvl and
guest_loglvl is set to all.
> >
> > I looked at the xm dmesg output and there''s nothing more than
what I already provided, aside from a bunch of commands from me poking at it.
> >
> > -Bruce
> >
> >
> > Thanks
> > --jyh
> >
> > From: Bruce Edge
> > [mailto:bruce.edge@gmail.com<mailto:bruce.edge@gmail.com>]
> > Sent: Tuesday, September 28, 2010 11:16 AM
> > To: Jiang, Yunhong
> > Cc: Konrad Rzeszutek Wilk;
> >
xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>
> >
> > Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts
> > on Nehalem
> >
> > On Mon, Sep 27, 2010 at 6:15 PM, Jiang, Yunhong
<yunhong.jiang@intel.com<mailto:yunhong.jiang@intel.com>> wrote:
> > Is the 07:0.0 your tachyon device? The VT-d fault is suspcious.
> >
> > Yes, there is 1 quad port card is this sytem:
> >
> > 07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> > 07:00.1 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> > 07:00.2 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> > 07:00.3 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> >
> >
> > Also is it possible to share the xen output?
> >
> > I attached the dom0 boot output. Let me know if you wanted something
else.
> >
> > Also, here''s the dom0 console output upon starting the VM:
This lockdep error started with the release of 2.6.32.21. Note that I''m
running the same  kernel for the domU and dom0.
> >
> > [ 1817.684097] ------------[ cut here ]------------ [ 1817.684113]
> > WARNING: at kernel/lockdep.c:2323
> > trace_hardirqs_on_caller+0x12f/0x190()
> > [ 1817.684119] Hardware name: ProLiant DL380 G6 [ 1817.684122]
> > Modules linked in: xt_physdev ipv6 osa_mfgdom0 xenfs xen_gntdev
> > fbcon tileblit font bitblit softcursor xen_evtchn xen_pciback radeon
> > ttm drm_kms_helper tun drm i2c_algo_bit ipmi_si i2c_core
ipmi_msghandler joydev serio_raw hpwdt hpilo bridge stp llc usbhid hid cciss
usb_storage [ 1817.684190] Pid: 11, comm: xenwatch Not tainted
2.6.32.21-xenoprof-1 #1 [ 1817.684195] Call Trace:
> > [ 1817.684197]  <IRQ>  [<ffffffff810aa18f>] ?
> > trace_hardirqs_on_caller+0x12f/0x190
> > [ 1817.684209]  [<ffffffff8106bed0>]
warn_slowpath_common+0x80/0xd0
> > [ 1817.684217]  [<ffffffff815f2b80>] ?
_spin_unlock_irq+0x30/0x40 [
> > 1817.684223]  [<ffffffff8106bf34>] warn_slowpath_null+0x14/0x20
[
> > 1817.684229]  [<ffffffff810aa18f>]
> > trace_hardirqs_on_caller+0x12f/0x190
> > [ 1817.684234]  [<ffffffff810aa1fd>] trace_hardirqs_on+0xd/0x10
[
> > 1817.684240]  [<ffffffff815f2b80>] _spin_unlock_irq+0x30/0x40 [
> > 1817.684266]  [<ffffffff813c4fc5>]
> > add_to_net_schedule_list_tail+0x85/0xd0
> > [ 1817.684271]  [<ffffffff813c6216>] netif_be_int+0x36/0x160 [
> > 1817.684278]  [<ffffffff810e10d0>] handle_IRQ_event+0x70/0x180 [
> > 1817.684284]  [<ffffffff810e36e9>] handle_edge_irq+0xc9/0x170 [
> > 1817.684291]  [<ffffffff813b8d7f>]
> > __xen_evtchn_do_upcall+0x1bf/0x1f0
> > [ 1817.684297]  [<ffffffff813b92fd>]
xen_evtchn_do_upcall+0x3d/0x60
> > [ 1817.684304]  [<ffffffff8101647e>]
> > xen_do_hypervisor_callback+0x1e/0x30
> > [ 1817.684308]  <EOI>  [<ffffffff8100940a>] ?
> > hypercall_page+0x40a/0x1010 [ 1817.684319]  [<ffffffff8100940a>]
?
> > hypercall_page+0x40a/0x1010 [ 1817.684325]  [<ffffffff813bce54>]
?
> > xb_write+0x1e4/0x290 [ 1817.684330]  [<ffffffff813bd8ca>] ?
> > xs_talkv+0x6a/0x1f0 [ 1817.684336]  [<ffffffff813bd8d8>] ?
> > xs_talkv+0x78/0x1f0 [ 1817.684341]  [<ffffffff813bdbcd>] ?
> > xs_single+0x4d/0x60 [ 1817.684346]  [<ffffffff813be502>] ?
> > xenbus_read+0x52/0x80 [ 1817.684352]  [<ffffffff813c87fc>] ?
> > frontend_changed+0x48c/0x770 [ 1817.684358] 
[<ffffffff813bf76d>] ?
> > xenbus_otherend_changed+0xdd/0x1b0
> > [ 1817.684365]  [<ffffffff8101122f>] ?
> > xen_restore_fl_direct_end+0x0/0x1 [ 1817.684371]
> > [<ffffffff810ac830>] ? lock_release+0xb0/0x230 [ 1817.684376] 
[<ffffffff813bfae0>] ?
> > frontend_changed+0x10/0x20 [ 1817.684382]  [<ffffffff813bd4f5>]
?
> > xenwatch_thread+0x55/0x160 [ 1817.684389]  [<ffffffff81093400>]
?
> > autoremove_wake_function+0x0/0x40 [ 1817.684394]
> > [<ffffffff813bd4a0>] ? xenwatch_thread+0x0/0x160 [ 1817.684400] 
[<ffffffff81093086>] ?
> > kthread+0x96/0xb0 [ 1817.684405]  [<ffffffff8101632a>] ?
> > child_rip+0xa/0x20 [ 1817.684410]  [<ffffffff81015c90>] ?
> > restore_args+0x0/0x30 [ 1817.684415]  [<ffffffff81016320>] ?
> > child_rip+0x0/0x20
> >
> > -Bruce
> >
> >
> >
> > Thanks
> > --jyh
> >
> > >-----Original Message-----
> > >From:
> >
>xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounces@lists.
> > >xensource.com>
> >
>[mailto:xen-devel-bounces@lists.xensource.com<mailto:xen-devel-boun
> > >ce s@lists.xensource.com>] On Behalf Of Bruce Edge
> > >Sent: Tuesday, September 28, 2010 7:54 AM
> > >To: Konrad Rzeszutek Wilk
> > >Cc:
> >
>xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>
> > >Subject: Re: [Xen-devel] pv-ops domU not working with MSI
> > >interrupts on Nehalem
> > >
> > >On Mon, Sep 27, 2010 at 12:54 PM, Konrad Rzeszutek Wilk
> >
><konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>> wrote:
> > >> On Mon, Sep 27, 2010 at 12:16:50PM -0700, Bruce Edge wrote:
> > >>> On Mon, Sep 27, 2010 at 10:24 AM, Konrad Rzeszutek Wilk
> > >>>
<konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>> wrote:
> > >>> >
> > >>> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge
wrote:
> > >>> > > One of our developers who is working on a
tachyon driver is
> > >>> > > complaining that the pvops domU kernel is not
working for
> > >>> > > these MSI interrupts.
> > >>> > > This is using the current head of xen/2.6.32.x
on both a
> > >>> > > single Nahelam 920 and a dual E5540. This
behavior is
> > >>> > > consistent with Xen 4.0.1, 4.0.2.rc1-pre and
4.1.
> > >>> > >
> > >>> > > Here are his comments:
> > >>> > >
> > >>> > > - the driver has no problem to enable msi
interrupt and
> > >>> > > request the interrupt through kernel functions
> > >>> > > pci_enable_msi & request_irq
> > >>> >
> > >>> > What shows up in the Xen console when you send the
''q'' key?
> > >>> > Does it show that the vector is assigned to the
appropiate guest?
> > >>>
> > >>> The Xen console q key shows that the domU is assigned:
> > >>>
> > >>> (XEN)     Interrupts { 32, 41-42, 47 }
> > >>
> > >> Aha!
> > >>
> > >>>
> > >>> but the domU thinks it has:
> > >>>
> > >>> 124/125/126/127
> > >>>
> > >>> Is there some mapping that''s taking place, or is
this plain wrong?
> > >>
> > >> That looks wrong. The IRQ numbers (even though they are MSI
> > >> vectors) are setup as IRQ numbers in the DomU guest. You
should
> > >> have seen
> > >>
> > >> 32:
> > >> 41:
> > >> 42:
> > >> 47:
> > >> in you /proc/interrupts on your DomU guest.
> > >>
> > >> I wonder what broke  - can you use
> >
>git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http:/
> > >/g it.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git>
> > >> devel/xen-pcifront-0.5 (or pv/pcifront-2.6.32)?
> > >
> > >Please forgive the git ignorance.
> > >
> > >Is this the right syntax?
> > >
> > >git clone
> >
>git://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.
> >
>32<http://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifron
> > >t-
> > >2.6.32>
> > >linux-2.6.32-pv-pcifront
> > >
> > >Initialized empty Git repository in
> >
>/import/kaan/bedge/src/xen/kernel/pv-ops/linux-2.6.32-pv-pcifront/.
> > >gi
> > >t/
> > >fatal: The remote end hung up unexpectedly
> > >
> > >Or:
> > >
> > > git clone
> > >
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http:
> > > // git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git>
> > >
> > >Initialized empty Git repository in
> > >/import/kaan/bedge/src/xen/kernel/pv-ops/xen/.git/
> > >remote: error: Could not read
> > >59eab2f8f04147c5aadc99f2034ca7e5b81e890f
> > >remote: fatal: Failed to traverse parents of commit
> > >979e121cb348add17ed8171bf447b27a3a9d1be3
> > >remote: aborting due to possible repository corruption on the
remote side.
> > >fatal: early EOF
> > >fatal: index-pack failed
> > >
> > >>
> > >> It has the latest pcifront driver but without the PVonHVM
> > >> enhancments so we can try to eliminate the PvONHVM logic out
of the picture.
> > >>
> > >>>
> > >>> >
> > >>> > > - the interrupt does happen. But the interrupt
service
> > >>> > > routine of tachyon driver doesn''t
detect any interrupt
> > >>> > > status related to this interrupt, which
inhibits the tachyon
> > >>> > > chip from coming on-line. And there are high
count of
> > >>> > > tachyon interrupt in /proc/interrupts
> > >>> >
> > >>> > Is it checking the PCI_STATUS_INTERRUPT or the
appropiate
> > >>> > register in the MMIO BAR?
> > >>> >
> > >>>
> > >>> The driver would check the appropriate register (tachyon
> > >>> registers) in the MMIO to determine the source of
interrupts.
> > >>
> > >> OK, so that isn''t it. Is there anything at these
vectors:
> > >> 7c, 7d, 7e, and 7f? When you use xen debug-keys
''i'' or ''q'' it
> > >> should give you an inkling what device this is set for.
> > >
> > >When I run a distro kernel in hvm mode, I get the expected irq
mappings:
> > >
> > >''i'' - Note 66 - 69
> > >(XEN)    IRQ:  66 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:3a
> > >type=PCI-MSI         status=00000010 in-flight=0
> > >domain-list=10:127(----),
> > >(XEN)    IRQ:  67 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:42
> > >type=PCI-MSI         status=00000010 in-flight=0
> > >domain-list=10:126(----),
> > >(XEN)    IRQ:  68 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:4a
> > >type=PCI-MSI         status=00000010 in-flight=0
> > >domain-list=10:125(----),
> > >(XEN)    IRQ:  69 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:52
> > >type=PCI-MSI         status=00000010 in-flight=0
> > >domain-list=10:124(----)
> > >
> > >
> > >''q''
> > >(XEN)     Interrupts { 32, 41-42, 47, 124-127 }
> > >
> > >
> > >The same data with pv-ops kernel shows:
> > >
> > >''i''
> > >IRQ numbers stop at 65, no 66 - 69 present:
> > >
> > >(XEN)    IRQ:  63 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:91
> > >type=PCI-MSI         status=00000010 in-flight=0
> > >domain-list=0:289(----),
> > >(XEN)    IRQ:  64 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:99
> > >type=PCI-MSI         status=00000002 mapped, unbound
> > >(XEN)    IRQ:  65 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:b1
> > >type=PCI-MSI         status=00000010 in-flight=0
> > >domain-list=0:287(----),
> > >(XEN) IO-APIC interrupt information:
> > >
> > >''q''
> > >(XEN)     Interrupts { 32, 41-42, 47 }
> > >
> > >>
> > >>>
> > >>> > >
> > >>> > > kaan-18-dpm:~# cat /proc/interrupts | grep TACH
> > >>> > >
> > >124:     760415          0          0          0          0
> > >    0
> > >>> > >          0          0          0          0    
0
> > >      0
> > >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
> > >>> > >
> > >125:     762234          0          0          0          0
> > >    0
> > >>> > >          0          0          0          0    
0
> > >      0
> > >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
> > >>> > >
> > >126:     764180          0          0          0          0
> > >    0
> > >>> > >          0          0          0          0    
0
> > >      0
> > >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
> > >>> > >
> > >127:     764164          0          0          0          0
> > >    0
> > >>> > >          0          0          0          0    
0
> > >      0
> > >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
> > >>> >
> > >>> > Can you provide the full dmesg output?
> > >>>
> > >>> Attached.
> > >>>
> > >>> Some possibly related messages on dom0 console:
> > >>>
> > >>> [ 1882.269778] pciback 0000:07:00.0: enabling device
(0000 ->
> > >>> 0003) [ 1882.269800] xen: registering gsi 32 triggering 0
> > >>> polarity
> > >>> 1 [ 1882.269827] xen_allocate_pirq: returning irq 32 for
gsi 32
> > >>> [ 1882.269834] xen: --> irq=32 [ 1882.269841] Already
setup the
> > >>> GSI
> > >>> :32 [ 1882.269847] pciback 0000:07:00.0: PCI INT A ->
GSI 32
> > >>> (level, low) -> IRQ 32 [ 1882.269866] pciback
0000:07:00.0:
> > >>> setting latency timer to 64 [ 1882.270463] pciback
0000:07:00.0:
> > >>> Driver tried to write to a read-only configuration space
field
> > >>> at offset 0x62, size 2. This may be harmless, but if you
have
> > >>> problems with your device:
> > >>
> > >> Uhhh, for that I think you need to do ''lspci -vvv
-xxx -s 07:00.00''
> > >> to find out what is at the configuration space. You could
enable
> > >> it using the permissive attribute.
> > >>
> > >>> [ 1882.270465] 1) see permissive attribute in sysfs [
> > >>> 1882.270467]
> > >>> 2) report problems to the xen-devel mailing list along
with
> > >>> details of your device obtained from lspci.
> > >>> [ 1882.270615]   alloc irq_desc for 478 on node 0
> > >>> [ 1882.270625]   alloc kstat_irqs on node 0
> > >>
> > >> So for 478: what do you see? xen-pciback I presume?
> > >>> [ 1882.348411] pciback 0000:07:00.1: enabling device
(0000 ->
> > >>> 0003) [ 1882.348433] xen: registering gsi 42 triggering 0
> > >>> polarity
> > >>> 1 [ 1882.348440] xen_allocate_pirq: returning irq 42 for
gsi 42
> > >>> [ 1882.348445] xen: --> irq=42 [ 1882.348472] Already
setup the
> > >>> GSI
> > >>> :42 [ 1882.348479] pciback 0000:07:00.1: PCI INT B ->
GSI 42
> > >>> (level, low) -> IRQ 42 [ 1882.348497] pciback
0000:07:00.1:
> > >>> setting latency timer to 64 [ 1882.349063] pciback
0000:07:00.1:
> > >>> Driver tried to write to a read-only configuration space
field
> > >>> at offset 0x62, size 2. This may be harmless, but if you
have
> > >>> problems with your device:
> > >>> [ 1882.349066] 1) see permissive attribute in sysfs [
> > >>> 1882.349067]
> > >>> 2) report problems to the xen-devel mailing list along
with
> > >>> details of your device obtained from lspci.
> > >>> [ 1882.349205]   alloc irq_desc for 477 on node 0
> > >>> [ 1882.349215]   alloc kstat_irqs on node 0
> > >>> [ 1882.402893] pciback 0000:07:00.2: enabling device
(0000 ->
> > >>> 0003) [ 1882.402908] xen: registering gsi 47 triggering 0
> > >>> polarity
> > >>> 1 [ 1882.402913] xen_allocate_pirq: returning irq 47 for
gsi 47
> > >>> [ 1882.402916] xen: --> irq=47 [ 1882.402921] Already
setup the
> > >>> GSI
> > >>> :47 [ 1882.402925] pciback 0000:07:00.2: PCI INT C ->
GSI 47
> > >>> (level, low) -> IRQ 47 [ 1882.402938] pciback
0000:07:00.2:
> > >>> setting latency timer to 64 [ 1882.403280] pciback
0000:07:00.2:
> > >>> Driver tried to write to a read-only configuration space
field
> > >>> at offset 0x62, size 2. This may be harmless, but if you
have
> > >>> problems with your device:
> > >>> [ 1882.403282] 1) see permissive attribute in sysfs [
> > >>> 1882.403282]
> > >>> 2) report problems to the xen-devel mailing list along
with
> > >>> details of your device obtained from lspci.
> > >>> [ 1882.403380]   alloc irq_desc for 476 on node 0
> > >>> [ 1882.403386]   alloc kstat_irqs on node 0
> > >>> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary
Pending
> > >>> Fault
> > >>> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device
> > >>> [07:00.0] fault addr e6f80000, iommu reg =
ffff82c3fff57000
> > >>> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
> > >>> (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf =
7:0.0 gmfn = e6f80
> > >>> (XEN)     root_entry = ffff83019ff70000
> > >>> (XEN)     root_entry[7] = 19cf52001
> > >>> (XEN)     context = ffff83019cf52000
> > >>> (XEN)     context[0] = 102_706dc005
> > >>> (XEN)     l4 = ffff8300706dc000
> > >>> (XEN)     l4_index = 0
> > >>> (XEN)     l4[0] = 706db003
> > >>> (XEN)     l3 = ffff8300706db000
> > >>> (XEN)     l3_index = 3
> > >>> (XEN)     l3[3] = 702b6003
> > >>> (XEN)     l2 = ffff8300702b6000
> > >>> (XEN)     l2_index = 137
> > >>> (XEN)     l2[137] = 0
> > >>> (XEN)     l2[137] not present
> > >>> (XEN) traps.c:466:d0 Unhandled nmi fault/trap [#2] on
VCPU 0
> > >>> [ec=0000]
> > >>
> > >> That is not good. What changed from your earlier emails that
this was triggered?
> > >
> > >Nothing
> > >> Or was it triggered all along?
> > >
> > >Yes, I just included it for completeness
> > >
> > >> What happens if you run the system without the iommu enabled?
> > >
> > >Haven''t tried yet. Will check that next.
> > >
> > >-Bruce
> > >
> > >_______________________________________________
> > >Xen-devel mailing list
> >
>Xen-devel@lists.xensource.com<mailto:Xen-devel@lists.xensource.com>
> > >http://lists.xensource.com/xen-devel
> >
> >
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Lin, Ray

2010-Sep-28 23:25 UTC

head link

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Here is how the memory is allocated for the DMA address in domU,

Non-NUMA:
get_free_pages((in_interrupt() ? GFP_ATOMIC : GFP_KERNEL)|GFP_DMA32, order))

NUMA:
alloc_pages_node(region->node, (in_interrupt() ? GFP_ATOMIC :
GFP_KERNEL)|GFP_DMA32, order)


Does this way work for pvops kernel ?


-Ray


-----Original Message-----
From: Lin, Ray
Sent: Tuesday, September 28, 2010 1:38 PM
To: ''Konrad Rzeszutek Wilk''
Cc: JBeulich@novell.com; xen-devel@lists.xensource.com; Jiang, Yunhong; Bruce
Edge
Subject: RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem


The physical address (0xc0049c) is not the BAR of the PCI device. It resides in
main memory. Is there any restriction to do the DMA Write to main memory in
pvops kernel ?


-Ray


-----Original Message-----
From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
Sent: Tuesday, September 28, 2010 1:15 PM
To: Lin, Ray
Cc: JBeulich@novell.com; xen-devel@lists.xensource.com; Jiang, Yunhong; Bruce
Edge
Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Tue, Sep 28, 2010 at 12:35:12PM -0600, Lin, Ray
wrote:>
> > (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
> > (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
> > fault addr c00000, iommu reg = ffff82c3fff57000
>
> The driver gets the physical addr 0xc0049c thru kernel function
virt_to_phys() and set the dma address of Tachyon chip with this address. This
address translation is also involved with SWIOTLB library. Is there any issue
related with SWIOTLB in pvops kernel ?
If the driver is using pci_map_page, which goes through the Xen-SWIOTLB library,
then it works correctly (the Xen-SWIOTLB does a PFN->MFN translation to give
you the bus address).

I presume the physical address (0xc0049c) is the BAR of the PCI device, right?

There are no issues with the Xen-SWIOTLB in the pvops kernel.
>
>
>
>
> -Ray
>
>
>
>
> -----Original Message-----
> From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
> Sent: Tuesday, September 28, 2010 9:19 AM
> To: Lin, Ray; JBeulich@novell.com
> Cc: Bruce Edge; Jiang, Yunhong; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts
> on Nehalem
>
> On Tue, Sep 28, 2010 at 10:08:57AM -0600, Lin, Ray wrote:
> >     I just checked the "xen dmesg". Look like DMA/iommu is
the root cause of this issue. In order to tell the source of interrupt, Tachyon
chip needs to do the DMA write to a dword memory location to indicate the source
of interrupt. What iommu option do you recommend to use ?
>
> Lets get Jan involved in this discussion.
>
> Jan, would some of your patches that inhibit the MSI write affect this in a
PV guest?
>
> >
> > (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
> > (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
> > fault addr c00000, iommu reg = ffff82c3fff57000
> > (XEN) DMAR:[fault reason 05h] PTE Write access is not set
> > (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn =
c00
> > (XEN)     root_entry = ffff83019ff70000
> > (XEN)     root_entry[7] = 19cf52001
> > (XEN)     context = ffff83019cf52000
> > (XEN)     context[0] = 102_706dc005
> > (XEN)     l4 = ffff8300706dc000
> > (XEN)     l4_index = 0
> > (XEN)     l4[0] = 706db003
> > (XEN)     l3 = ffff8300706db000
> > (XEN)     l3_index = 0
> > (XEN)     l3[0] = 706da003
> > (XEN)     l2 = ffff8300706da000
> > (XEN)     l2_index = 6
> > (XEN)     l2[6] = 0
> >
> >
> > -Ray
> >
> >
> > ________________________________
> > From: xen-devel-bounces@lists.xensource.com
> > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Bruce
> > Edge
> > Sent: Monday, September 27, 2010 9:46 PM
> > To: Jiang, Yunhong
> > Cc: xen-devel@lists.xensource.com; Konrad Rzeszutek Wilk
> > Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts
> > on Nehalem
> >
> > On Mon, Sep 27, 2010 at 8:26 PM, Jiang, Yunhong
<yunhong.jiang@intel.com<mailto:yunhong.jiang@intel.com>> wrote:
> > "xm dmesg" should gives xen''s boot log, and
sometimes it contain some helpful information, I think, especially loglvl and
guest_loglvl is set to all.
> >
> > I looked at the xm dmesg output and there''s nothing more than
what I already provided, aside from a bunch of commands from me poking at it.
> >
> > -Bruce
> >
> >
> > Thanks
> > --jyh
> >
> > From: Bruce Edge
> > [mailto:bruce.edge@gmail.com<mailto:bruce.edge@gmail.com>]
> > Sent: Tuesday, September 28, 2010 11:16 AM
> > To: Jiang, Yunhong
> > Cc: Konrad Rzeszutek Wilk;
> >
xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>
> >
> > Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts
> > on Nehalem
> >
> > On Mon, Sep 27, 2010 at 6:15 PM, Jiang, Yunhong
<yunhong.jiang@intel.com<mailto:yunhong.jiang@intel.com>> wrote:
> > Is the 07:0.0 your tachyon device? The VT-d fault is suspcious.
> >
> > Yes, there is 1 quad port card is this sytem:
> >
> > 07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> > 07:00.1 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> > 07:00.2 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> > 07:00.3 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> >
> >
> > Also is it possible to share the xen output?
> >
> > I attached the dom0 boot output. Let me know if you wanted something
else.
> >
> > Also, here''s the dom0 console output upon starting the VM:
This lockdep error started with the release of 2.6.32.21. Note that I''m
running the same  kernel for the domU and dom0.
> >
> > [ 1817.684097] ------------[ cut here ]------------ [ 1817.684113]
> > WARNING: at kernel/lockdep.c:2323
> > trace_hardirqs_on_caller+0x12f/0x190()
> > [ 1817.684119] Hardware name: ProLiant DL380 G6 [ 1817.684122]
> > Modules linked in: xt_physdev ipv6 osa_mfgdom0 xenfs xen_gntdev
> > fbcon tileblit font bitblit softcursor xen_evtchn xen_pciback radeon
> > ttm drm_kms_helper tun drm i2c_algo_bit ipmi_si i2c_core
ipmi_msghandler joydev serio_raw hpwdt hpilo bridge stp llc usbhid hid cciss
usb_storage [ 1817.684190] Pid: 11, comm: xenwatch Not tainted
2.6.32.21-xenoprof-1 #1 [ 1817.684195] Call Trace:
> > [ 1817.684197]  <IRQ>  [<ffffffff810aa18f>] ?
> > trace_hardirqs_on_caller+0x12f/0x190
> > [ 1817.684209]  [<ffffffff8106bed0>]
warn_slowpath_common+0x80/0xd0
> > [ 1817.684217]  [<ffffffff815f2b80>] ?
_spin_unlock_irq+0x30/0x40 [
> > 1817.684223]  [<ffffffff8106bf34>] warn_slowpath_null+0x14/0x20
[
> > 1817.684229]  [<ffffffff810aa18f>]
> > trace_hardirqs_on_caller+0x12f/0x190
> > [ 1817.684234]  [<ffffffff810aa1fd>] trace_hardirqs_on+0xd/0x10
[
> > 1817.684240]  [<ffffffff815f2b80>] _spin_unlock_irq+0x30/0x40 [
> > 1817.684266]  [<ffffffff813c4fc5>]
> > add_to_net_schedule_list_tail+0x85/0xd0
> > [ 1817.684271]  [<ffffffff813c6216>] netif_be_int+0x36/0x160 [
> > 1817.684278]  [<ffffffff810e10d0>] handle_IRQ_event+0x70/0x180 [
> > 1817.684284]  [<ffffffff810e36e9>] handle_edge_irq+0xc9/0x170 [
> > 1817.684291]  [<ffffffff813b8d7f>]
> > __xen_evtchn_do_upcall+0x1bf/0x1f0
> > [ 1817.684297]  [<ffffffff813b92fd>]
xen_evtchn_do_upcall+0x3d/0x60
> > [ 1817.684304]  [<ffffffff8101647e>]
> > xen_do_hypervisor_callback+0x1e/0x30
> > [ 1817.684308]  <EOI>  [<ffffffff8100940a>] ?
> > hypercall_page+0x40a/0x1010 [ 1817.684319]  [<ffffffff8100940a>]
?
> > hypercall_page+0x40a/0x1010 [ 1817.684325]  [<ffffffff813bce54>]
?
> > xb_write+0x1e4/0x290 [ 1817.684330]  [<ffffffff813bd8ca>] ?
> > xs_talkv+0x6a/0x1f0 [ 1817.684336]  [<ffffffff813bd8d8>] ?
> > xs_talkv+0x78/0x1f0 [ 1817.684341]  [<ffffffff813bdbcd>] ?
> > xs_single+0x4d/0x60 [ 1817.684346]  [<ffffffff813be502>] ?
> > xenbus_read+0x52/0x80 [ 1817.684352]  [<ffffffff813c87fc>] ?
> > frontend_changed+0x48c/0x770 [ 1817.684358] 
[<ffffffff813bf76d>] ?
> > xenbus_otherend_changed+0xdd/0x1b0
> > [ 1817.684365]  [<ffffffff8101122f>] ?
> > xen_restore_fl_direct_end+0x0/0x1 [ 1817.684371]
> > [<ffffffff810ac830>] ? lock_release+0xb0/0x230 [ 1817.684376] 
[<ffffffff813bfae0>] ?
> > frontend_changed+0x10/0x20 [ 1817.684382]  [<ffffffff813bd4f5>]
?
> > xenwatch_thread+0x55/0x160 [ 1817.684389]  [<ffffffff81093400>]
?
> > autoremove_wake_function+0x0/0x40 [ 1817.684394]
> > [<ffffffff813bd4a0>] ? xenwatch_thread+0x0/0x160 [ 1817.684400] 
[<ffffffff81093086>] ?
> > kthread+0x96/0xb0 [ 1817.684405]  [<ffffffff8101632a>] ?
> > child_rip+0xa/0x20 [ 1817.684410]  [<ffffffff81015c90>] ?
> > restore_args+0x0/0x30 [ 1817.684415]  [<ffffffff81016320>] ?
> > child_rip+0x0/0x20
> >
> > -Bruce
> >
> >
> >
> > Thanks
> > --jyh
> >
> > >-----Original Message-----
> > >From:
> >
>xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounces@lists.
> > >xensource.com>
> >
>[mailto:xen-devel-bounces@lists.xensource.com<mailto:xen-devel-boun
> > >ce s@lists.xensource.com>] On Behalf Of Bruce Edge
> > >Sent: Tuesday, September 28, 2010 7:54 AM
> > >To: Konrad Rzeszutek Wilk
> > >Cc:
> >
>xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>
> > >Subject: Re: [Xen-devel] pv-ops domU not working with MSI
> > >interrupts on Nehalem
> > >
> > >On Mon, Sep 27, 2010 at 12:54 PM, Konrad Rzeszutek Wilk
> >
><konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>> wrote:
> > >> On Mon, Sep 27, 2010 at 12:16:50PM -0700, Bruce Edge wrote:
> > >>> On Mon, Sep 27, 2010 at 10:24 AM, Konrad Rzeszutek Wilk
> > >>>
<konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>> wrote:
> > >>> >
> > >>> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge
wrote:
> > >>> > > One of our developers who is working on a
tachyon driver is
> > >>> > > complaining that the pvops domU kernel is not
working for
> > >>> > > these MSI interrupts.
> > >>> > > This is using the current head of xen/2.6.32.x
on both a
> > >>> > > single Nahelam 920 and a dual E5540. This
behavior is
> > >>> > > consistent with Xen 4.0.1, 4.0.2.rc1-pre and
4.1.
> > >>> > >
> > >>> > > Here are his comments:
> > >>> > >
> > >>> > > - the driver has no problem to enable msi
interrupt and
> > >>> > > request the interrupt through kernel functions
> > >>> > > pci_enable_msi & request_irq
> > >>> >
> > >>> > What shows up in the Xen console when you send the
''q'' key?
> > >>> > Does it show that the vector is assigned to the
appropiate guest?
> > >>>
> > >>> The Xen console q key shows that the domU is assigned:
> > >>>
> > >>> (XEN)     Interrupts { 32, 41-42, 47 }
> > >>
> > >> Aha!
> > >>
> > >>>
> > >>> but the domU thinks it has:
> > >>>
> > >>> 124/125/126/127
> > >>>
> > >>> Is there some mapping that''s taking place, or is
this plain wrong?
> > >>
> > >> That looks wrong. The IRQ numbers (even though they are MSI
> > >> vectors) are setup as IRQ numbers in the DomU guest. You
should
> > >> have seen
> > >>
> > >> 32:
> > >> 41:
> > >> 42:
> > >> 47:
> > >> in you /proc/interrupts on your DomU guest.
> > >>
> > >> I wonder what broke  - can you use
> >
>git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http:/
> > >/g it.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git>
> > >> devel/xen-pcifront-0.5 (or pv/pcifront-2.6.32)?
> > >
> > >Please forgive the git ignorance.
> > >
> > >Is this the right syntax?
> > >
> > >git clone
> >
>git://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.
> >
>32<http://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifron
> > >t-
> > >2.6.32>
> > >linux-2.6.32-pv-pcifront
> > >
> > >Initialized empty Git repository in
> >
>/import/kaan/bedge/src/xen/kernel/pv-ops/linux-2.6.32-pv-pcifront/.
> > >gi
> > >t/
> > >fatal: The remote end hung up unexpectedly
> > >
> > >Or:
> > >
> > > git clone
> > >
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http:
> > > // git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git>
> > >
> > >Initialized empty Git repository in
> > >/import/kaan/bedge/src/xen/kernel/pv-ops/xen/.git/
> > >remote: error: Could not read
> > >59eab2f8f04147c5aadc99f2034ca7e5b81e890f
> > >remote: fatal: Failed to traverse parents of commit
> > >979e121cb348add17ed8171bf447b27a3a9d1be3
> > >remote: aborting due to possible repository corruption on the
remote side.
> > >fatal: early EOF
> > >fatal: index-pack failed
> > >
> > >>
> > >> It has the latest pcifront driver but without the PVonHVM
> > >> enhancments so we can try to eliminate the PvONHVM logic out
of the picture.
> > >>
> > >>>
> > >>> >
> > >>> > > - the interrupt does happen. But the interrupt
service
> > >>> > > routine of tachyon driver doesn''t
detect any interrupt
> > >>> > > status related to this interrupt, which
inhibits the tachyon
> > >>> > > chip from coming on-line. And there are high
count of
> > >>> > > tachyon interrupt in /proc/interrupts
> > >>> >
> > >>> > Is it checking the PCI_STATUS_INTERRUPT or the
appropiate
> > >>> > register in the MMIO BAR?
> > >>> >
> > >>>
> > >>> The driver would check the appropriate register (tachyon
> > >>> registers) in the MMIO to determine the source of
interrupts.
> > >>
> > >> OK, so that isn''t it. Is there anything at these
vectors:
> > >> 7c, 7d, 7e, and 7f? When you use xen debug-keys
''i'' or ''q'' it
> > >> should give you an inkling what device this is set for.
> > >
> > >When I run a distro kernel in hvm mode, I get the expected irq
mappings:
> > >
> > >''i'' - Note 66 - 69
> > >(XEN)    IRQ:  66 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:3a
> > >type=PCI-MSI         status=00000010 in-flight=0
> > >domain-list=10:127(----),
> > >(XEN)    IRQ:  67 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:42
> > >type=PCI-MSI         status=00000010 in-flight=0
> > >domain-list=10:126(----),
> > >(XEN)    IRQ:  68 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:4a
> > >type=PCI-MSI         status=00000010 in-flight=0
> > >domain-list=10:125(----),
> > >(XEN)    IRQ:  69 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:52
> > >type=PCI-MSI         status=00000010 in-flight=0
> > >domain-list=10:124(----)
> > >
> > >
> > >''q''
> > >(XEN)     Interrupts { 32, 41-42, 47, 124-127 }
> > >
> > >
> > >The same data with pv-ops kernel shows:
> > >
> > >''i''
> > >IRQ numbers stop at 65, no 66 - 69 present:
> > >
> > >(XEN)    IRQ:  63 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:91
> > >type=PCI-MSI         status=00000010 in-flight=0
> > >domain-list=0:289(----),
> > >(XEN)    IRQ:  64 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:99
> > >type=PCI-MSI         status=00000002 mapped, unbound
> > >(XEN)    IRQ:  65 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:b1
> > >type=PCI-MSI         status=00000010 in-flight=0
> > >domain-list=0:287(----),
> > >(XEN) IO-APIC interrupt information:
> > >
> > >''q''
> > >(XEN)     Interrupts { 32, 41-42, 47 }
> > >
> > >>
> > >>>
> > >>> > >
> > >>> > > kaan-18-dpm:~# cat /proc/interrupts | grep TACH
> > >>> > >
> > >124:     760415          0          0          0          0
> > >    0
> > >>> > >          0          0          0          0    
0
> > >      0
> > >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
> > >>> > >
> > >125:     762234          0          0          0          0
> > >    0
> > >>> > >          0          0          0          0    
0
> > >      0
> > >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
> > >>> > >
> > >126:     764180          0          0          0          0
> > >    0
> > >>> > >          0          0          0          0    
0
> > >      0
> > >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
> > >>> > >
> > >127:     764164          0          0          0          0
> > >    0
> > >>> > >          0          0          0          0    
0
> > >      0
> > >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
> > >>> >
> > >>> > Can you provide the full dmesg output?
> > >>>
> > >>> Attached.
> > >>>
> > >>> Some possibly related messages on dom0 console:
> > >>>
> > >>> [ 1882.269778] pciback 0000:07:00.0: enabling device
(0000 ->
> > >>> 0003) [ 1882.269800] xen: registering gsi 32 triggering 0
> > >>> polarity
> > >>> 1 [ 1882.269827] xen_allocate_pirq: returning irq 32 for
gsi 32
> > >>> [ 1882.269834] xen: --> irq=32 [ 1882.269841] Already
setup the
> > >>> GSI
> > >>> :32 [ 1882.269847] pciback 0000:07:00.0: PCI INT A ->
GSI 32
> > >>> (level, low) -> IRQ 32 [ 1882.269866] pciback
0000:07:00.0:
> > >>> setting latency timer to 64 [ 1882.270463] pciback
0000:07:00.0:
> > >>> Driver tried to write to a read-only configuration space
field
> > >>> at offset 0x62, size 2. This may be harmless, but if you
have
> > >>> problems with your device:
> > >>
> > >> Uhhh, for that I think you need to do ''lspci -vvv
-xxx -s 07:00.00''
> > >> to find out what is at the configuration space. You could
enable
> > >> it using the permissive attribute.
> > >>
> > >>> [ 1882.270465] 1) see permissive attribute in sysfs [
> > >>> 1882.270467]
> > >>> 2) report problems to the xen-devel mailing list along
with
> > >>> details of your device obtained from lspci.
> > >>> [ 1882.270615]   alloc irq_desc for 478 on node 0
> > >>> [ 1882.270625]   alloc kstat_irqs on node 0
> > >>
> > >> So for 478: what do you see? xen-pciback I presume?
> > >>> [ 1882.348411] pciback 0000:07:00.1: enabling device
(0000 ->
> > >>> 0003) [ 1882.348433] xen: registering gsi 42 triggering 0
> > >>> polarity
> > >>> 1 [ 1882.348440] xen_allocate_pirq: returning irq 42 for
gsi 42
> > >>> [ 1882.348445] xen: --> irq=42 [ 1882.348472] Already
setup the
> > >>> GSI
> > >>> :42 [ 1882.348479] pciback 0000:07:00.1: PCI INT B ->
GSI 42
> > >>> (level, low) -> IRQ 42 [ 1882.348497] pciback
0000:07:00.1:
> > >>> setting latency timer to 64 [ 1882.349063] pciback
0000:07:00.1:
> > >>> Driver tried to write to a read-only configuration space
field
> > >>> at offset 0x62, size 2. This may be harmless, but if you
have
> > >>> problems with your device:
> > >>> [ 1882.349066] 1) see permissive attribute in sysfs [
> > >>> 1882.349067]
> > >>> 2) report problems to the xen-devel mailing list along
with
> > >>> details of your device obtained from lspci.
> > >>> [ 1882.349205]   alloc irq_desc for 477 on node 0
> > >>> [ 1882.349215]   alloc kstat_irqs on node 0
> > >>> [ 1882.402893] pciback 0000:07:00.2: enabling device
(0000 ->
> > >>> 0003) [ 1882.402908] xen: registering gsi 47 triggering 0
> > >>> polarity
> > >>> 1 [ 1882.402913] xen_allocate_pirq: returning irq 47 for
gsi 47
> > >>> [ 1882.402916] xen: --> irq=47 [ 1882.402921] Already
setup the
> > >>> GSI
> > >>> :47 [ 1882.402925] pciback 0000:07:00.2: PCI INT C ->
GSI 47
> > >>> (level, low) -> IRQ 47 [ 1882.402938] pciback
0000:07:00.2:
> > >>> setting latency timer to 64 [ 1882.403280] pciback
0000:07:00.2:
> > >>> Driver tried to write to a read-only configuration space
field
> > >>> at offset 0x62, size 2. This may be harmless, but if you
have
> > >>> problems with your device:
> > >>> [ 1882.403282] 1) see permissive attribute in sysfs [
> > >>> 1882.403282]
> > >>> 2) report problems to the xen-devel mailing list along
with
> > >>> details of your device obtained from lspci.
> > >>> [ 1882.403380]   alloc irq_desc for 476 on node 0
> > >>> [ 1882.403386]   alloc kstat_irqs on node 0
> > >>> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary
Pending
> > >>> Fault
> > >>> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device
> > >>> [07:00.0] fault addr e6f80000, iommu reg =
ffff82c3fff57000
> > >>> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
> > >>> (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf =
7:0.0 gmfn = e6f80
> > >>> (XEN)     root_entry = ffff83019ff70000
> > >>> (XEN)     root_entry[7] = 19cf52001
> > >>> (XEN)     context = ffff83019cf52000
> > >>> (XEN)     context[0] = 102_706dc005
> > >>> (XEN)     l4 = ffff8300706dc000
> > >>> (XEN)     l4_index = 0
> > >>> (XEN)     l4[0] = 706db003
> > >>> (XEN)     l3 = ffff8300706db000
> > >>> (XEN)     l3_index = 3
> > >>> (XEN)     l3[3] = 702b6003
> > >>> (XEN)     l2 = ffff8300702b6000
> > >>> (XEN)     l2_index = 137
> > >>> (XEN)     l2[137] = 0
> > >>> (XEN)     l2[137] not present
> > >>> (XEN) traps.c:466:d0 Unhandled nmi fault/trap [#2] on
VCPU 0
> > >>> [ec=0000]
> > >>
> > >> That is not good. What changed from your earlier emails that
this was triggered?
> > >
> > >Nothing
> > >> Or was it triggered all along?
> > >
> > >Yes, I just included it for completeness
> > >
> > >> What happens if you run the system without the iommu enabled?
> > >
> > >Haven''t tried yet. Will check that next.
> > >
> > >-Bruce
> > >
> > >_______________________________________________
> > >Xen-devel mailing list
> >
>Xen-devel@lists.xensource.com<mailto:Xen-devel@lists.xensource.com>
> > >http://lists.xensource.com/xen-devel
> >
> >
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk

2010-Sep-28 23:57 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Tue, Sep 28, 2010 at 05:25:48PM -0600, Lin, Ray
wrote:> 
> 
> Here is how the memory is allocated for the DMA address in domU,
> 
> Non-NUMA:
> get_free_pages((in_interrupt() ? GFP_ATOMIC : GFP_KERNEL)|GFP_DMA32,
order))
> 
> NUMA:
> alloc_pages_node(region->node, (in_interrupt() ? GFP_ATOMIC :
GFP_KERNEL)|GFP_DMA32, order)
> 
> 
> Does this way work for pvops kernel ?
Sure, as long as you physical address you provide to the device for DMA has
gone through the PCI API - meaning you did pci_map_page or
pci_map_<whatever>
> 
> 
> -Ray
> 
> 
> -----Original Message-----
> From: Lin, Ray
> Sent: Tuesday, September 28, 2010 1:38 PM
> To: ''Konrad Rzeszutek Wilk''
> Cc: JBeulich@novell.com; xen-devel@lists.xensource.com; Jiang, Yunhong;
Bruce Edge
> Subject: RE: [Xen-devel] pv-ops domU not working with MSI interrupts on
Nehalem
> 
> 
> The physical address (0xc0049c) is not the BAR of the PCI device. It
resides in main memory. Is there any restriction to do the DMA Write to main
memory in pvops kernel ?
> 
> 
> -Ray
> 
> 
> -----Original Message-----
> From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
> Sent: Tuesday, September 28, 2010 1:15 PM
> To: Lin, Ray
> Cc: JBeulich@novell.com; xen-devel@lists.xensource.com; Jiang, Yunhong;
Bruce Edge
> Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on
Nehalem
> 
> On Tue, Sep 28, 2010 at 12:35:12PM -0600, Lin, Ray wrote:
> >
> > > (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending
Fault
> > > (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device
[07:00.0]
> > > fault addr c00000, iommu reg = ffff82c3fff57000
> >
> > The driver gets the physical addr 0xc0049c thru kernel function
virt_to_phys() and set the dma address of Tachyon chip with this address. This
address translation is also involved with SWIOTLB library. Is there any issue
related with SWIOTLB in pvops kernel ?
> 
> If the driver is using pci_map_page, which goes through the Xen-SWIOTLB
library, then it works correctly (the Xen-SWIOTLB does a PFN->MFN translation
to give you the bus address).
> 
> I presume the physical address (0xc0049c) is the BAR of the PCI device,
right?
> 
> There are no issues with the Xen-SWIOTLB in the pvops kernel.
> 
> >
> >
> >
> >
> > -Ray
> >
> >
> >
> >
> > -----Original Message-----
> > From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
> > Sent: Tuesday, September 28, 2010 9:19 AM
> > To: Lin, Ray; JBeulich@novell.com
> > Cc: Bruce Edge; Jiang, Yunhong; xen-devel@lists.xensource.com
> > Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts
> > on Nehalem
> >
> > On Tue, Sep 28, 2010 at 10:08:57AM -0600, Lin, Ray wrote:
> > >     I just checked the "xen dmesg". Look like DMA/iommu
is the root cause of this issue. In order to tell the source of interrupt,
Tachyon chip needs to do the DMA write to a dword memory location to indicate
the source of interrupt. What iommu option do you recommend to use ?
> >
> > Lets get Jan involved in this discussion.
> >
> > Jan, would some of your patches that inhibit the MSI write affect this
in a PV guest?
> >
> > >
> > > (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending
Fault
> > > (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device
[07:00.0]
> > > fault addr c00000, iommu reg = ffff82c3fff57000
> > > (XEN) DMAR:[fault reason 05h] PTE Write access is not set
> > > (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0
gmfn = c00
> > > (XEN)     root_entry = ffff83019ff70000
> > > (XEN)     root_entry[7] = 19cf52001
> > > (XEN)     context = ffff83019cf52000
> > > (XEN)     context[0] = 102_706dc005
> > > (XEN)     l4 = ffff8300706dc000
> > > (XEN)     l4_index = 0
> > > (XEN)     l4[0] = 706db003
> > > (XEN)     l3 = ffff8300706db000
> > > (XEN)     l3_index = 0
> > > (XEN)     l3[0] = 706da003
> > > (XEN)     l2 = ffff8300706da000
> > > (XEN)     l2_index = 6
> > > (XEN)     l2[6] = 0
> > >
> > >
> > > -Ray
> > >
> > >
> > > ________________________________
> > > From: xen-devel-bounces@lists.xensource.com
> > > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Bruce
> > > Edge
> > > Sent: Monday, September 27, 2010 9:46 PM
> > > To: Jiang, Yunhong
> > > Cc: xen-devel@lists.xensource.com; Konrad Rzeszutek Wilk
> > > Subject: Re: [Xen-devel] pv-ops domU not working with MSI
interrupts
> > > on Nehalem
> > >
> > > On Mon, Sep 27, 2010 at 8:26 PM, Jiang, Yunhong
<yunhong.jiang@intel.com<mailto:yunhong.jiang@intel.com>> wrote:
> > > "xm dmesg" should gives xen''s boot log, and
sometimes it contain some helpful information, I think, especially loglvl and
guest_loglvl is set to all.
> > >
> > > I looked at the xm dmesg output and there''s nothing more
than what I already provided, aside from a bunch of commands from me poking at
it.
> > >
> > > -Bruce
> > >
> > >
> > > Thanks
> > > --jyh
> > >
> > > From: Bruce Edge
> > > [mailto:bruce.edge@gmail.com<mailto:bruce.edge@gmail.com>]
> > > Sent: Tuesday, September 28, 2010 11:16 AM
> > > To: Jiang, Yunhong
> > > Cc: Konrad Rzeszutek Wilk;
> > >
xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>
> > >
> > > Subject: Re: [Xen-devel] pv-ops domU not working with MSI
interrupts
> > > on Nehalem
> > >
> > > On Mon, Sep 27, 2010 at 6:15 PM, Jiang, Yunhong
<yunhong.jiang@intel.com<mailto:yunhong.jiang@intel.com>> wrote:
> > > Is the 07:0.0 your tachyon device? The VT-d fault is suspcious.
> > >
> > > Yes, there is 1 quad port card is this sytem:
> > >
> > > 07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> > > 07:00.1 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> > > 07:00.2 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> > > 07:00.3 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
> > >
> > >
> > > Also is it possible to share the xen output?
> > >
> > > I attached the dom0 boot output. Let me know if you wanted
something else.
> > >
> > > Also, here''s the dom0 console output upon starting the
VM: This lockdep error started with the release of 2.6.32.21. Note that
I''m running the same  kernel for the domU and dom0.
> > >
> > > [ 1817.684097] ------------[ cut here ]------------ [
1817.684113]
> > > WARNING: at kernel/lockdep.c:2323
> > > trace_hardirqs_on_caller+0x12f/0x190()
> > > [ 1817.684119] Hardware name: ProLiant DL380 G6 [ 1817.684122]
> > > Modules linked in: xt_physdev ipv6 osa_mfgdom0 xenfs xen_gntdev
> > > fbcon tileblit font bitblit softcursor xen_evtchn xen_pciback
radeon
> > > ttm drm_kms_helper tun drm i2c_algo_bit ipmi_si i2c_core
ipmi_msghandler joydev serio_raw hpwdt hpilo bridge stp llc usbhid hid cciss
usb_storage [ 1817.684190] Pid: 11, comm: xenwatch Not tainted
2.6.32.21-xenoprof-1 #1 [ 1817.684195] Call Trace:
> > > [ 1817.684197]  <IRQ>  [<ffffffff810aa18f>] ?
> > > trace_hardirqs_on_caller+0x12f/0x190
> > > [ 1817.684209]  [<ffffffff8106bed0>]
warn_slowpath_common+0x80/0xd0
> > > [ 1817.684217]  [<ffffffff815f2b80>] ?
_spin_unlock_irq+0x30/0x40 [
> > > 1817.684223]  [<ffffffff8106bf34>]
warn_slowpath_null+0x14/0x20 [
> > > 1817.684229]  [<ffffffff810aa18f>]
> > > trace_hardirqs_on_caller+0x12f/0x190
> > > [ 1817.684234]  [<ffffffff810aa1fd>]
trace_hardirqs_on+0xd/0x10 [
> > > 1817.684240]  [<ffffffff815f2b80>]
_spin_unlock_irq+0x30/0x40 [
> > > 1817.684266]  [<ffffffff813c4fc5>]
> > > add_to_net_schedule_list_tail+0x85/0xd0
> > > [ 1817.684271]  [<ffffffff813c6216>]
netif_be_int+0x36/0x160 [
> > > 1817.684278]  [<ffffffff810e10d0>]
handle_IRQ_event+0x70/0x180 [
> > > 1817.684284]  [<ffffffff810e36e9>]
handle_edge_irq+0xc9/0x170 [
> > > 1817.684291]  [<ffffffff813b8d7f>]
> > > __xen_evtchn_do_upcall+0x1bf/0x1f0
> > > [ 1817.684297]  [<ffffffff813b92fd>]
xen_evtchn_do_upcall+0x3d/0x60
> > > [ 1817.684304]  [<ffffffff8101647e>]
> > > xen_do_hypervisor_callback+0x1e/0x30
> > > [ 1817.684308]  <EOI>  [<ffffffff8100940a>] ?
> > > hypercall_page+0x40a/0x1010 [ 1817.684319] 
[<ffffffff8100940a>] ?
> > > hypercall_page+0x40a/0x1010 [ 1817.684325] 
[<ffffffff813bce54>] ?
> > > xb_write+0x1e4/0x290 [ 1817.684330]  [<ffffffff813bd8ca>] ?
> > > xs_talkv+0x6a/0x1f0 [ 1817.684336]  [<ffffffff813bd8d8>] ?
> > > xs_talkv+0x78/0x1f0 [ 1817.684341]  [<ffffffff813bdbcd>] ?
> > > xs_single+0x4d/0x60 [ 1817.684346]  [<ffffffff813be502>] ?
> > > xenbus_read+0x52/0x80 [ 1817.684352]  [<ffffffff813c87fc>]
?
> > > frontend_changed+0x48c/0x770 [ 1817.684358] 
[<ffffffff813bf76d>] ?
> > > xenbus_otherend_changed+0xdd/0x1b0
> > > [ 1817.684365]  [<ffffffff8101122f>] ?
> > > xen_restore_fl_direct_end+0x0/0x1 [ 1817.684371]
> > > [<ffffffff810ac830>] ? lock_release+0xb0/0x230 [
1817.684376]  [<ffffffff813bfae0>] ?
> > > frontend_changed+0x10/0x20 [ 1817.684382] 
[<ffffffff813bd4f5>] ?
> > > xenwatch_thread+0x55/0x160 [ 1817.684389] 
[<ffffffff81093400>] ?
> > > autoremove_wake_function+0x0/0x40 [ 1817.684394]
> > > [<ffffffff813bd4a0>] ? xenwatch_thread+0x0/0x160 [
1817.684400]  [<ffffffff81093086>] ?
> > > kthread+0x96/0xb0 [ 1817.684405]  [<ffffffff8101632a>] ?
> > > child_rip+0xa/0x20 [ 1817.684410]  [<ffffffff81015c90>] ?
> > > restore_args+0x0/0x30 [ 1817.684415]  [<ffffffff81016320>]
?
> > > child_rip+0x0/0x20
> > >
> > > -Bruce
> > >
> > >
> > >
> > > Thanks
> > > --jyh
> > >
> > > >-----Original Message-----
> > > >From:
> > >
>xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounces@lists.
> > > >xensource.com>
> > >
>[mailto:xen-devel-bounces@lists.xensource.com<mailto:xen-devel-boun
> > > >ce s@lists.xensource.com>] On Behalf Of Bruce Edge
> > > >Sent: Tuesday, September 28, 2010 7:54 AM
> > > >To: Konrad Rzeszutek Wilk
> > > >Cc:
> > >
>xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>
> > > >Subject: Re: [Xen-devel] pv-ops domU not working with MSI
> > > >interrupts on Nehalem
> > > >
> > > >On Mon, Sep 27, 2010 at 12:54 PM, Konrad Rzeszutek Wilk
> > >
><konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>> wrote:
> > > >> On Mon, Sep 27, 2010 at 12:16:50PM -0700, Bruce Edge
wrote:
> > > >>> On Mon, Sep 27, 2010 at 10:24 AM, Konrad Rzeszutek
Wilk
> > > >>>
<konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>> wrote:
> > > >>> >
> > > >>> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce
Edge wrote:
> > > >>> > > One of our developers who is working on a
tachyon driver is
> > > >>> > > complaining that the pvops domU kernel is
not working for
> > > >>> > > these MSI interrupts.
> > > >>> > > This is using the current head of
xen/2.6.32.x on both a
> > > >>> > > single Nahelam 920 and a dual E5540. This
behavior is
> > > >>> > > consistent with Xen 4.0.1, 4.0.2.rc1-pre
and 4.1.
> > > >>> > >
> > > >>> > > Here are his comments:
> > > >>> > >
> > > >>> > > - the driver has no problem to enable msi
interrupt and
> > > >>> > > request the interrupt through kernel
functions
> > > >>> > > pci_enable_msi & request_irq
> > > >>> >
> > > >>> > What shows up in the Xen console when you send
the ''q'' key?
> > > >>> > Does it show that the vector is assigned to the
appropiate guest?
> > > >>>
> > > >>> The Xen console q key shows that the domU is
assigned:
> > > >>>
> > > >>> (XEN)     Interrupts { 32, 41-42, 47 }
> > > >>
> > > >> Aha!
> > > >>
> > > >>>
> > > >>> but the domU thinks it has:
> > > >>>
> > > >>> 124/125/126/127
> > > >>>
> > > >>> Is there some mapping that''s taking place,
or is this plain wrong?
> > > >>
> > > >> That looks wrong. The IRQ numbers (even though they are
MSI
> > > >> vectors) are setup as IRQ numbers in the DomU guest. You
should
> > > >> have seen
> > > >>
> > > >> 32:
> > > >> 41:
> > > >> 42:
> > > >> 47:
> > > >> in you /proc/interrupts on your DomU guest.
> > > >>
> > > >> I wonder what broke  - can you use
> > >
>git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http:/
> > > >/g it.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git>
> > > >> devel/xen-pcifront-0.5 (or pv/pcifront-2.6.32)?
> > > >
> > > >Please forgive the git ignorance.
> > > >
> > > >Is this the right syntax?
> > > >
> > > >git clone
> > >
>git://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.
> > >
>32<http://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifron
> > > >t-
> > > >2.6.32>
> > > >linux-2.6.32-pv-pcifront
> > > >
> > > >Initialized empty Git repository in
> > >
>/import/kaan/bedge/src/xen/kernel/pv-ops/linux-2.6.32-pv-pcifront/.
> > > >gi
> > > >t/
> > > >fatal: The remote end hung up unexpectedly
> > > >
> > > >Or:
> > > >
> > > > git clone
> > > >
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http:
> > > > //
git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git>
> > > >
> > > >Initialized empty Git repository in
> > > >/import/kaan/bedge/src/xen/kernel/pv-ops/xen/.git/
> > > >remote: error: Could not read
> > > >59eab2f8f04147c5aadc99f2034ca7e5b81e890f
> > > >remote: fatal: Failed to traverse parents of commit
> > > >979e121cb348add17ed8171bf447b27a3a9d1be3
> > > >remote: aborting due to possible repository corruption on the
remote side.
> > > >fatal: early EOF
> > > >fatal: index-pack failed
> > > >
> > > >>
> > > >> It has the latest pcifront driver but without the
PVonHVM
> > > >> enhancments so we can try to eliminate the PvONHVM logic
out of the picture.
> > > >>
> > > >>>
> > > >>> >
> > > >>> > > - the interrupt does happen. But the
interrupt service
> > > >>> > > routine of tachyon driver doesn''t
detect any interrupt
> > > >>> > > status related to this interrupt, which
inhibits the tachyon
> > > >>> > > chip from coming on-line. And there are
high count of
> > > >>> > > tachyon interrupt in /proc/interrupts
> > > >>> >
> > > >>> > Is it checking the PCI_STATUS_INTERRUPT or the
appropiate
> > > >>> > register in the MMIO BAR?
> > > >>> >
> > > >>>
> > > >>> The driver would check the appropriate register
(tachyon
> > > >>> registers) in the MMIO to determine the source of
interrupts.
> > > >>
> > > >> OK, so that isn''t it. Is there anything at
these vectors:
> > > >> 7c, 7d, 7e, and 7f? When you use xen debug-keys
''i'' or ''q'' it
> > > >> should give you an inkling what device this is set for.
> > > >
> > > >When I run a distro kernel in hvm mode, I get the expected
irq mappings:
> > > >
> > > >''i'' - Note 66 - 69
> > > >(XEN)    IRQ:  66
affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:3a
> > > >type=PCI-MSI         status=00000010 in-flight=0
> > > >domain-list=10:127(----),
> > > >(XEN)    IRQ:  67
affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:42
> > > >type=PCI-MSI         status=00000010 in-flight=0
> > > >domain-list=10:126(----),
> > > >(XEN)    IRQ:  68
affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:4a
> > > >type=PCI-MSI         status=00000010 in-flight=0
> > > >domain-list=10:125(----),
> > > >(XEN)    IRQ:  69
affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:52
> > > >type=PCI-MSI         status=00000010 in-flight=0
> > > >domain-list=10:124(----)
> > > >
> > > >
> > > >''q''
> > > >(XEN)     Interrupts { 32, 41-42, 47, 124-127 }
> > > >
> > > >
> > > >The same data with pv-ops kernel shows:
> > > >
> > > >''i''
> > > >IRQ numbers stop at 65, no 66 - 69 present:
> > > >
> > > >(XEN)    IRQ:  63
affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:91
> > > >type=PCI-MSI         status=00000010 in-flight=0
> > > >domain-list=0:289(----),
> > > >(XEN)    IRQ:  64
affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:99
> > > >type=PCI-MSI         status=00000002 mapped, unbound
> > > >(XEN)    IRQ:  65
affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:b1
> > > >type=PCI-MSI         status=00000010 in-flight=0
> > > >domain-list=0:287(----),
> > > >(XEN) IO-APIC interrupt information:
> > > >
> > > >''q''
> > > >(XEN)     Interrupts { 32, 41-42, 47 }
> > > >
> > > >>
> > > >>>
> > > >>> > >
> > > >>> > > kaan-18-dpm:~# cat /proc/interrupts | grep
TACH
> > > >>> > >
> > > >124:     760415          0          0          0          0
> > > >    0
> > > >>> > >          0          0          0         
0          0
> > > >      0
> > > >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
> > > >>> > >
> > > >125:     762234          0          0          0          0
> > > >    0
> > > >>> > >          0          0          0         
0          0
> > > >      0
> > > >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
> > > >>> > >
> > > >126:     764180          0          0          0          0
> > > >    0
> > > >>> > >          0          0          0         
0          0
> > > >      0
> > > >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
> > > >>> > >
> > > >127:     764164          0          0          0          0
> > > >    0
> > > >>> > >          0          0          0         
0          0
> > > >      0
> > > >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
> > > >>> >
> > > >>> > Can you provide the full dmesg output?
> > > >>>
> > > >>> Attached.
> > > >>>
> > > >>> Some possibly related messages on dom0 console:
> > > >>>
> > > >>> [ 1882.269778] pciback 0000:07:00.0: enabling device
(0000 ->
> > > >>> 0003) [ 1882.269800] xen: registering gsi 32
triggering 0
> > > >>> polarity
> > > >>> 1 [ 1882.269827] xen_allocate_pirq: returning irq 32
for gsi 32
> > > >>> [ 1882.269834] xen: --> irq=32 [ 1882.269841]
Already setup the
> > > >>> GSI
> > > >>> :32 [ 1882.269847] pciback 0000:07:00.0: PCI INT A
-> GSI 32
> > > >>> (level, low) -> IRQ 32 [ 1882.269866] pciback
0000:07:00.0:
> > > >>> setting latency timer to 64 [ 1882.270463] pciback
0000:07:00.0:
> > > >>> Driver tried to write to a read-only configuration
space field
> > > >>> at offset 0x62, size 2. This may be harmless, but if
you have
> > > >>> problems with your device:
> > > >>
> > > >> Uhhh, for that I think you need to do ''lspci
-vvv -xxx -s 07:00.00''
> > > >> to find out what is at the configuration space. You
could enable
> > > >> it using the permissive attribute.
> > > >>
> > > >>> [ 1882.270465] 1) see permissive attribute in sysfs
[
> > > >>> 1882.270467]
> > > >>> 2) report problems to the xen-devel mailing list
along with
> > > >>> details of your device obtained from lspci.
> > > >>> [ 1882.270615]   alloc irq_desc for 478 on node 0
> > > >>> [ 1882.270625]   alloc kstat_irqs on node 0
> > > >>
> > > >> So for 478: what do you see? xen-pciback I presume?
> > > >>> [ 1882.348411] pciback 0000:07:00.1: enabling device
(0000 ->
> > > >>> 0003) [ 1882.348433] xen: registering gsi 42
triggering 0
> > > >>> polarity
> > > >>> 1 [ 1882.348440] xen_allocate_pirq: returning irq 42
for gsi 42
> > > >>> [ 1882.348445] xen: --> irq=42 [ 1882.348472]
Already setup the
> > > >>> GSI
> > > >>> :42 [ 1882.348479] pciback 0000:07:00.1: PCI INT B
-> GSI 42
> > > >>> (level, low) -> IRQ 42 [ 1882.348497] pciback
0000:07:00.1:
> > > >>> setting latency timer to 64 [ 1882.349063] pciback
0000:07:00.1:
> > > >>> Driver tried to write to a read-only configuration
space field
> > > >>> at offset 0x62, size 2. This may be harmless, but if
you have
> > > >>> problems with your device:
> > > >>> [ 1882.349066] 1) see permissive attribute in sysfs
[
> > > >>> 1882.349067]
> > > >>> 2) report problems to the xen-devel mailing list
along with
> > > >>> details of your device obtained from lspci.
> > > >>> [ 1882.349205]   alloc irq_desc for 477 on node 0
> > > >>> [ 1882.349215]   alloc kstat_irqs on node 0
> > > >>> [ 1882.402893] pciback 0000:07:00.2: enabling device
(0000 ->
> > > >>> 0003) [ 1882.402908] xen: registering gsi 47
triggering 0
> > > >>> polarity
> > > >>> 1 [ 1882.402913] xen_allocate_pirq: returning irq 47
for gsi 47
> > > >>> [ 1882.402916] xen: --> irq=47 [ 1882.402921]
Already setup the
> > > >>> GSI
> > > >>> :47 [ 1882.402925] pciback 0000:07:00.2: PCI INT C
-> GSI 47
> > > >>> (level, low) -> IRQ 47 [ 1882.402938] pciback
0000:07:00.2:
> > > >>> setting latency timer to 64 [ 1882.403280] pciback
0000:07:00.2:
> > > >>> Driver tried to write to a read-only configuration
space field
> > > >>> at offset 0x62, size 2. This may be harmless, but if
you have
> > > >>> problems with your device:
> > > >>> [ 1882.403282] 1) see permissive attribute in sysfs
[
> > > >>> 1882.403282]
> > > >>> 2) report problems to the xen-devel mailing list
along with
> > > >>> details of your device obtained from lspci.
> > > >>> [ 1882.403380]   alloc irq_desc for 476 on node 0
> > > >>> [ 1882.403386]   alloc kstat_irqs on node 0
> > > >>> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary
Pending
> > > >>> Fault
> > > >>> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request
device
> > > >>> [07:00.0] fault addr e6f80000, iommu reg =
ffff82c3fff57000
> > > >>> (XEN) DMAR:[fault reason 05h] PTE Write access is
not set
> > > >>> (XEN) print_vtd_entries: iommu = ffff83019fffa370
bdf = 7:0.0 gmfn = e6f80
> > > >>> (XEN)     root_entry = ffff83019ff70000
> > > >>> (XEN)     root_entry[7] = 19cf52001
> > > >>> (XEN)     context = ffff83019cf52000
> > > >>> (XEN)     context[0] = 102_706dc005
> > > >>> (XEN)     l4 = ffff8300706dc000
> > > >>> (XEN)     l4_index = 0
> > > >>> (XEN)     l4[0] = 706db003
> > > >>> (XEN)     l3 = ffff8300706db000
> > > >>> (XEN)     l3_index = 3
> > > >>> (XEN)     l3[3] = 702b6003
> > > >>> (XEN)     l2 = ffff8300702b6000
> > > >>> (XEN)     l2_index = 137
> > > >>> (XEN)     l2[137] = 0
> > > >>> (XEN)     l2[137] not present
> > > >>> (XEN) traps.c:466:d0 Unhandled nmi fault/trap [#2]
on VCPU 0
> > > >>> [ec=0000]
> > > >>
> > > >> That is not good. What changed from your earlier emails
that this was triggered?
> > > >
> > > >Nothing
> > > >> Or was it triggered all along?
> > > >
> > > >Yes, I just included it for completeness
> > > >
> > > >> What happens if you run the system without the iommu
enabled?
> > > >
> > > >Haven''t tried yet. Will check that next.
> > > >
> > > >-Bruce
> > > >
> > > >_______________________________________________
> > > >Xen-devel mailing list
> > >
>Xen-devel@lists.xensource.com<mailto:Xen-devel@lists.xensource.com>
> > > >http://lists.xensource.com/xen-devel
> > >
> > >
> 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Jiang, Yunhong

2010-Sep-29 00:53 UTC

head link

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

>-----Original Message-----
>From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
>Sent: Wednesday, September 29, 2010 12:19 AM
>To: Lin, Ray; JBeulich@novell.com
>Cc: Bruce Edge; Jiang, Yunhong; xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on
Nehalem
>
>On Tue, Sep 28, 2010 at 10:08:57AM -0600, Lin, Ray wrote:
>>     I just checked the "xen dmesg". Look like DMA/iommu is
the root cause of
>this issue. In order to tell the source of interrupt, Tachyon chip needs to
do the DMA
>write to a dword memory location to indicate the source of interrupt. What
iommu
>option do you recommend to use ?
>
>Lets get Jan involved in this discussion.
>
>Jan, would some of your patches that inhibit the MSI write affect this
>in a PV guest?
As far as I can tell, this patch should not cause issue to VT-d side, instead,
it''s more about access from CPU.
Can you try to pass a option to xen as "iommu=off", and check the
result?

Thanks
--jyh
>
>>
>> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
>> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
fault addr
>c00000, iommu reg = ffff82c3fff57000
>> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
>> (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn =
c00
>> (XEN)     root_entry = ffff83019ff70000
>> (XEN)     root_entry[7] = 19cf52001
>> (XEN)     context = ffff83019cf52000
>> (XEN)     context[0] = 102_706dc005
>> (XEN)     l4 = ffff8300706dc000
>> (XEN)     l4_index = 0
>> (XEN)     l4[0] = 706db003
>> (XEN)     l3 = ffff8300706db000
>> (XEN)     l3_index = 0
>> (XEN)     l3[0] = 706da003
>> (XEN)     l2 = ffff8300706da000
>> (XEN)     l2_index = 6
>> (XEN)     l2[6] = 0
>>
>>
>> -Ray
>>
>>
>> ________________________________
>> From: xen-devel-bounces@lists.xensource.com
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Bruce Edge
>> Sent: Monday, September 27, 2010 9:46 PM
>> To: Jiang, Yunhong
>> Cc: xen-devel@lists.xensource.com; Konrad Rzeszutek Wilk
>> Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on
Nehalem
>>
>> On Mon, Sep 27, 2010 at 8:26 PM, Jiang, Yunhong
><yunhong.jiang@intel.com<mailto:yunhong.jiang@intel.com>> wrote:
>> "xm dmesg" should gives xen''s boot log, and
sometimes it contain some helpful
>information, I think, especially loglvl and guest_loglvl is set to all.
>>
>> I looked at the xm dmesg output and there''s nothing more than
what I already
>provided, aside from a bunch of commands from me poking at it.
>>
>> -Bruce
>>
>>
>> Thanks
>> --jyh
>>
>> From: Bruce Edge
>[mailto:bruce.edge@gmail.com<mailto:bruce.edge@gmail.com>]
>> Sent: Tuesday, September 28, 2010 11:16 AM
>> To: Jiang, Yunhong
>> Cc: Konrad Rzeszutek Wilk;
>xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>
>>
>> Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on
Nehalem
>>
>> On Mon, Sep 27, 2010 at 6:15 PM, Jiang, Yunhong
><yunhong.jiang@intel.com<mailto:yunhong.jiang@intel.com>> wrote:
>> Is the 07:0.0 your tachyon device? The VT-d fault is suspcious.
>>
>> Yes, there is 1 quad port card is this sytem:
>>
>> 07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
>> 07:00.1 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
>> 07:00.2 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
>> 07:00.3 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
>>
>>
>> Also is it possible to share the xen output?
>>
>> I attached the dom0 boot output. Let me know if you wanted something
else.
>>
>> Also, here''s the dom0 console output upon starting the VM:
This lockdep error
>started with the release of 2.6.32.21. Note that I''m running the
same  kernel for
>the domU and dom0.
>>
>> [ 1817.684097] ------------[ cut here ]------------
>> [ 1817.684113] WARNING: at kernel/lockdep.c:2323
>trace_hardirqs_on_caller+0x12f/0x190()
>> [ 1817.684119] Hardware name: ProLiant DL380 G6
>> [ 1817.684122] Modules linked in: xt_physdev ipv6 osa_mfgdom0 xenfs
xen_gntdev
>fbcon tileblit font bitblit softcursor xen_evtchn xen_pciback radeon ttm
>drm_kms_helper tun drm i2c_algo_bit ipmi_si i2c_core ipmi_msghandler joydev
>serio_raw hpwdt hpilo bridge stp llc usbhid hid cciss usb_storage
>> [ 1817.684190] Pid: 11, comm: xenwatch Not tainted 2.6.32.21-xenoprof-1
#1
>> [ 1817.684195] Call Trace:
>> [ 1817.684197]  <IRQ>  [<ffffffff810aa18f>] ?
>trace_hardirqs_on_caller+0x12f/0x190
>> [ 1817.684209]  [<ffffffff8106bed0>]
warn_slowpath_common+0x80/0xd0
>> [ 1817.684217]  [<ffffffff815f2b80>] ? _spin_unlock_irq+0x30/0x40
>> [ 1817.684223]  [<ffffffff8106bf34>] warn_slowpath_null+0x14/0x20
>> [ 1817.684229]  [<ffffffff810aa18f>]
trace_hardirqs_on_caller+0x12f/0x190
>> [ 1817.684234]  [<ffffffff810aa1fd>] trace_hardirqs_on+0xd/0x10
>> [ 1817.684240]  [<ffffffff815f2b80>] _spin_unlock_irq+0x30/0x40
>> [ 1817.684266]  [<ffffffff813c4fc5>]
add_to_net_schedule_list_tail+0x85/0xd0
>> [ 1817.684271]  [<ffffffff813c6216>] netif_be_int+0x36/0x160
>> [ 1817.684278]  [<ffffffff810e10d0>] handle_IRQ_event+0x70/0x180
>> [ 1817.684284]  [<ffffffff810e36e9>] handle_edge_irq+0xc9/0x170
>> [ 1817.684291]  [<ffffffff813b8d7f>]
__xen_evtchn_do_upcall+0x1bf/0x1f0
>> [ 1817.684297]  [<ffffffff813b92fd>]
xen_evtchn_do_upcall+0x3d/0x60
>> [ 1817.684304]  [<ffffffff8101647e>]
xen_do_hypervisor_callback+0x1e/0x30
>> [ 1817.684308]  <EOI>  [<ffffffff8100940a>] ?
hypercall_page+0x40a/0x1010
>> [ 1817.684319]  [<ffffffff8100940a>] ?
hypercall_page+0x40a/0x1010
>> [ 1817.684325]  [<ffffffff813bce54>] ? xb_write+0x1e4/0x290
>> [ 1817.684330]  [<ffffffff813bd8ca>] ? xs_talkv+0x6a/0x1f0
>> [ 1817.684336]  [<ffffffff813bd8d8>] ? xs_talkv+0x78/0x1f0
>> [ 1817.684341]  [<ffffffff813bdbcd>] ? xs_single+0x4d/0x60
>> [ 1817.684346]  [<ffffffff813be502>] ? xenbus_read+0x52/0x80
>> [ 1817.684352]  [<ffffffff813c87fc>] ?
frontend_changed+0x48c/0x770
>> [ 1817.684358]  [<ffffffff813bf76d>] ?
xenbus_otherend_changed+0xdd/0x1b0
>> [ 1817.684365]  [<ffffffff8101122f>] ?
xen_restore_fl_direct_end+0x0/0x1
>> [ 1817.684371]  [<ffffffff810ac830>] ? lock_release+0xb0/0x230
>> [ 1817.684376]  [<ffffffff813bfae0>] ? frontend_changed+0x10/0x20
>> [ 1817.684382]  [<ffffffff813bd4f5>] ? xenwatch_thread+0x55/0x160
>> [ 1817.684389]  [<ffffffff81093400>] ?
autoremove_wake_function+0x0/0x40
>> [ 1817.684394]  [<ffffffff813bd4a0>] ? xenwatch_thread+0x0/0x160
>> [ 1817.684400]  [<ffffffff81093086>] ? kthread+0x96/0xb0
>> [ 1817.684405]  [<ffffffff8101632a>] ? child_rip+0xa/0x20
>> [ 1817.684410]  [<ffffffff81015c90>] ? restore_args+0x0/0x30
>> [ 1817.684415]  [<ffffffff81016320>] ? child_rip+0x0/0x20
>>
>> -Bruce
>>
>>
>>
>> Thanks
>> --jyh
>>
>> >-----Original Message-----
>> >From:
>xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounces@lists.xensourc
>e.com>
>>
>[mailto:xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounces@list
>s.xensource.com>] On Behalf Of Bruce Edge
>> >Sent: Tuesday, September 28, 2010 7:54 AM
>> >To: Konrad Rzeszutek Wilk
>> >Cc:
xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>
>> >Subject: Re: [Xen-devel] pv-ops domU not working with MSI
interrupts on
>Nehalem
>> >
>> >On Mon, Sep 27, 2010 at 12:54 PM, Konrad Rzeszutek Wilk
>> ><konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>>
wrote:
>> >> On Mon, Sep 27, 2010 at 12:16:50PM -0700, Bruce Edge wrote:
>> >>> On Mon, Sep 27, 2010 at 10:24 AM, Konrad Rzeszutek Wilk
>> >>>
<konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>> wrote:
>> >>> >
>> >>> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge
wrote:
>> >>> > > One of our developers who is working on a
tachyon driver is
>> >>> > > complaining that the pvops domU kernel is not
working for these MSI
>> >>> > > interrupts.
>> >>> > > This is using the current head of xen/2.6.32.x
on both a single
>> >>> > > Nahelam 920 and a dual E5540. This behavior is
consistent with Xen
>> >>> > > 4.0.1, 4.0.2.rc1-pre and 4.1.
>> >>> > >
>> >>> > > Here are his comments:
>> >>> > >
>> >>> > > - the driver has no problem to enable msi
interrupt and request the
>> >>> > > interrupt through kernel functions
pci_enable_msi & request_irq
>> >>> >
>> >>> > What shows up in the Xen console when you send the
''q'' key? Does it
>> >>> > show that the vector is assigned to the appropiate
guest?
>> >>>
>> >>> The Xen console q key shows that the domU is assigned:
>> >>>
>> >>> (XEN)     Interrupts { 32, 41-42, 47 }
>> >>
>> >> Aha!
>> >>
>> >>>
>> >>> but the domU thinks it has:
>> >>>
>> >>> 124/125/126/127
>> >>>
>> >>> Is there some mapping that''s taking place, or is
this plain wrong?
>> >>
>> >> That looks wrong. The IRQ numbers (even though they are MSI
vectors) are
>> >> setup as IRQ numbers in the DomU guest. You should have seen
>> >>
>> >> 32:
>> >> 41:
>> >> 42:
>> >> 47:
>> >> in you /proc/interrupts on your DomU guest.
>> >>
>> >> I wonder what broke  - can you use
>>
>git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http://git.kernel.org
>/pub/scm/linux/kernel/git/konrad/xen.git>
>> >> devel/xen-pcifront-0.5 (or pv/pcifront-2.6.32)?
>> >
>> >Please forgive the git ignorance.
>> >
>> >Is this the right syntax?
>> >
>> >git clone
>git://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.32<http://git.ke
>rnel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.32>
>> >linux-2.6.32-pv-pcifront
>> >
>> >Initialized empty Git repository in
>>
>/import/kaan/bedge/src/xen/kernel/pv-ops/linux-2.6.32-pv-pcifront/.git/
>> >fatal: The remote end hung up unexpectedly
>> >
>> >Or:
>> >
>> > git clone
>git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http://git.kernel.org/p
>ub/scm/linux/kernel/git/konrad/xen.git>
>> >
>> >Initialized empty Git repository in
>> >/import/kaan/bedge/src/xen/kernel/pv-ops/xen/.git/
>> >remote: error: Could not read
59eab2f8f04147c5aadc99f2034ca7e5b81e890f
>> >remote: fatal: Failed to traverse parents of commit
>> >979e121cb348add17ed8171bf447b27a3a9d1be3
>> >remote: aborting due to possible repository corruption on the
remote side.
>> >fatal: early EOF
>> >fatal: index-pack failed
>> >
>> >>
>> >> It has the latest pcifront driver but without the PVonHVM
enhancments
>> >> so we can try to eliminate the PvONHVM logic out of the
picture.
>> >>
>> >>>
>> >>> >
>> >>> > > - the interrupt does happen. But the interrupt
service routine of
>> >>> > > tachyon driver doesn''t detect any
interrupt status related to this
>> >>> > > interrupt, which inhibits the tachyon chip from
coming on-line. And
>> >>> > > there are high count of tachyon interrupt in
/proc/interrupts
>> >>> >
>> >>> > Is it checking the PCI_STATUS_INTERRUPT or the
appropiate register
>> >>> > in the MMIO BAR?
>> >>> >
>> >>>
>> >>> The driver would check the appropriate register (tachyon
registers) in
>> >>> the MMIO to determine the source of interrupts.
>> >>
>> >> OK, so that isn''t it. Is there anything at these
vectors:
>> >> 7c, 7d, 7e, and 7f? When you use xen debug-keys
''i'' or ''q'' it should give you
>> >> an inkling what device this is set for.
>> >
>> >When I run a distro kernel in hvm mode, I get the expected irq
mappings:
>> >
>> >''i'' - Note 66 - 69
>> >(XEN)    IRQ:  66 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:3a
>> >type=PCI-MSI         status=00000010 in-flight=0
>> >domain-list=10:127(----),
>> >(XEN)    IRQ:  67 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:42
>> >type=PCI-MSI         status=00000010 in-flight=0
>> >domain-list=10:126(----),
>> >(XEN)    IRQ:  68 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:4a
>> >type=PCI-MSI         status=00000010 in-flight=0
>> >domain-list=10:125(----),
>> >(XEN)    IRQ:  69 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:52
>> >type=PCI-MSI         status=00000010 in-flight=0
>> >domain-list=10:124(----)
>> >
>> >
>> >''q''
>> >(XEN)     Interrupts { 32, 41-42, 47, 124-127 }
>> >
>> >
>> >The same data with pv-ops kernel shows:
>> >
>> >''i''
>> >IRQ numbers stop at 65, no 66 - 69 present:
>> >
>> >(XEN)    IRQ:  63 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:91
>> >type=PCI-MSI         status=00000010 in-flight=0
>> >domain-list=0:289(----),
>> >(XEN)    IRQ:  64 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:99
>> >type=PCI-MSI         status=00000002 mapped, unbound
>> >(XEN)    IRQ:  65 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:b1
>> >type=PCI-MSI         status=00000010 in-flight=0
>> >domain-list=0:287(----),
>> >(XEN) IO-APIC interrupt information:
>> >
>> >''q''
>> >(XEN)     Interrupts { 32, 41-42, 47 }
>> >
>> >>
>> >>>
>> >>> > >
>> >>> > > kaan-18-dpm:~# cat /proc/interrupts | grep TACH
>> >>> > >
>> >124:     760415          0          0          0          0
>> >    0
>> >>> > >          0          0          0          0     
0
>> >      0
>> >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
>> >>> > >
>> >125:     762234          0          0          0          0
>> >    0
>> >>> > >          0          0          0          0     
0
>> >      0
>> >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
>> >>> > >
>> >126:     764180          0          0          0          0
>> >    0
>> >>> > >          0          0          0          0     
0
>> >      0
>> >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
>> >>> > >
>> >127:     764164          0          0          0          0
>> >    0
>> >>> > >          0          0          0          0     
0
>> >      0
>> >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
>> >>> >
>> >>> > Can you provide the full dmesg output?
>> >>>
>> >>> Attached.
>> >>>
>> >>> Some possibly related messages on dom0 console:
>> >>>
>> >>> [ 1882.269778] pciback 0000:07:00.0: enabling device (0000
-> 0003)
>> >>> [ 1882.269800] xen: registering gsi 32 triggering 0
polarity 1
>> >>> [ 1882.269827] xen_allocate_pirq: returning irq 32 for gsi
32
>> >>> [ 1882.269834] xen: --> irq=32
>> >>> [ 1882.269841] Already setup the GSI :32
>> >>> [ 1882.269847] pciback 0000:07:00.0: PCI INT A -> GSI
32 (level, low) -> IRQ 32
>> >>> [ 1882.269866] pciback 0000:07:00.0: setting latency timer
to 64
>> >>> [ 1882.270463] pciback 0000:07:00.0: Driver tried to write
to a
>> >>> read-only configuration space field at offset 0x62, size
2. This may
>> >>> be harmless, but if you have problems with your device:
>> >>
>> >> Uhhh, for that I think you need to do ''lspci -vvv
-xxx -s 07:00.00''
>> >> to find out what is at the configuration space. You could
enable
>> >> it using the permissive attribute.
>> >>
>> >>> [ 1882.270465] 1) see permissive attribute in sysfs
>> >>> [ 1882.270467] 2) report problems to the xen-devel mailing
list along
>> >>> with details of your device obtained from lspci.
>> >>> [ 1882.270615]   alloc irq_desc for 478 on node 0
>> >>> [ 1882.270625]   alloc kstat_irqs on node 0
>> >>
>> >> So for 478: what do you see? xen-pciback I presume?
>> >>> [ 1882.348411] pciback 0000:07:00.1: enabling device (0000
-> 0003)
>> >>> [ 1882.348433] xen: registering gsi 42 triggering 0
polarity 1
>> >>> [ 1882.348440] xen_allocate_pirq: returning irq 42 for gsi
42
>> >>> [ 1882.348445] xen: --> irq=42
>> >>> [ 1882.348472] Already setup the GSI :42
>> >>> [ 1882.348479] pciback 0000:07:00.1: PCI INT B -> GSI
42 (level, low) -> IRQ 42
>> >>> [ 1882.348497] pciback 0000:07:00.1: setting latency timer
to 64
>> >>> [ 1882.349063] pciback 0000:07:00.1: Driver tried to write
to a
>> >>> read-only configuration space field at offset 0x62, size
2. This may
>> >>> be harmless, but if you have problems with your device:
>> >>> [ 1882.349066] 1) see permissive attribute in sysfs
>> >>> [ 1882.349067] 2) report problems to the xen-devel mailing
list along
>> >>> with details of your device obtained from lspci.
>> >>> [ 1882.349205]   alloc irq_desc for 477 on node 0
>> >>> [ 1882.349215]   alloc kstat_irqs on node 0
>> >>> [ 1882.402893] pciback 0000:07:00.2: enabling device (0000
-> 0003)
>> >>> [ 1882.402908] xen: registering gsi 47 triggering 0
polarity 1
>> >>> [ 1882.402913] xen_allocate_pirq: returning irq 47 for gsi
47
>> >>> [ 1882.402916] xen: --> irq=47
>> >>> [ 1882.402921] Already setup the GSI :47
>> >>> [ 1882.402925] pciback 0000:07:00.2: PCI INT C -> GSI
47 (level, low) -> IRQ 47
>> >>> [ 1882.402938] pciback 0000:07:00.2: setting latency timer
to 64
>> >>> [ 1882.403280] pciback 0000:07:00.2: Driver tried to write
to a
>> >>> read-only configuration space field at offset 0x62, size
2. This may
>> >>> be harmless, but if you have problems with your device:
>> >>> [ 1882.403282] 1) see permissive attribute in sysfs
>> >>> [ 1882.403282] 2) report problems to the xen-devel mailing
list along
>> >>> with details of your device obtained from lspci.
>> >>> [ 1882.403380]   alloc irq_desc for 476 on node 0
>> >>> [ 1882.403386]   alloc kstat_irqs on node 0
>> >>> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary
Pending Fault
>> >>> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device
[07:00.0]
>> >>> fault addr e6f80000, iommu reg = ffff82c3fff57000
>> >>> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
>> >>> (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf =
7:0.0 gmfn = e6f80
>> >>> (XEN)     root_entry = ffff83019ff70000
>> >>> (XEN)     root_entry[7] = 19cf52001
>> >>> (XEN)     context = ffff83019cf52000
>> >>> (XEN)     context[0] = 102_706dc005
>> >>> (XEN)     l4 = ffff8300706dc000
>> >>> (XEN)     l4_index = 0
>> >>> (XEN)     l4[0] = 706db003
>> >>> (XEN)     l3 = ffff8300706db000
>> >>> (XEN)     l3_index = 3
>> >>> (XEN)     l3[3] = 702b6003
>> >>> (XEN)     l2 = ffff8300702b6000
>> >>> (XEN)     l2_index = 137
>> >>> (XEN)     l2[137] = 0
>> >>> (XEN)     l2[137] not present
>> >>> (XEN) traps.c:466:d0 Unhandled nmi fault/trap [#2] on VCPU
0 [ec=0000]
>> >>
>> >> That is not good. What changed from your earlier emails that
this was
>triggered?
>> >
>> >Nothing
>> >> Or was it triggered all along?
>> >
>> >Yes, I just included it for completeness
>> >
>> >> What happens if you run the system without the iommu enabled?
>> >
>> >Haven''t tried yet. Will check that next.
>> >
>> >-Bruce
>> >
>> >_______________________________________________
>> >Xen-devel mailing list
>>
>Xen-devel@lists.xensource.com<mailto:Xen-devel@lists.xensource.com>
>> >http://lists.xensource.com/xen-devel
>>
>>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Lin, Ray

2010-Sep-30 16:30 UTC

head link

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

It doesn''t help with "iommu=off". Here is the kernel command
line I set.

 (XEN) Command line: dummy=dummy dom0_mem=1024M dom0_max_vcpus=1
dom0_vcpus_pin=true iommu=off,passthrough,no-intremap loglvl=all
loglvl_guest=all loglevl=10 debug apic=on apic_verbosity=verbose
extra_guest_irqs=80 com1=115200,8n1 console=com1 console_to_ring noirqbalance
xen-pciback.permissive acpi=force numa=on


-Ray

-----Original Message-----
From: Jiang, Yunhong [mailto:yunhong.jiang@intel.com]
Sent: Tuesday, September 28, 2010 5:54 PM
To: Konrad Rzeszutek Wilk; Lin, Ray; JBeulich@novell.com
Cc: Bruce Edge; xen-devel@lists.xensource.com
Subject: RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

>-----Original Message-----
>From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com]
>Sent: Wednesday, September 29, 2010 12:19 AM
>To: Lin, Ray; JBeulich@novell.com
>Cc: Bruce Edge; Jiang, Yunhong; xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on
>Nehalem
>
>On Tue, Sep 28, 2010 at 10:08:57AM -0600, Lin, Ray wrote:
>>     I just checked the "xen dmesg". Look like DMA/iommu is
the root
>> cause of
>this issue. In order to tell the source of interrupt, Tachyon chip
>needs to do the DMA write to a dword memory location to indicate the
>source of interrupt. What iommu option do you recommend to use ?
>
>Lets get Jan involved in this discussion.
>
>Jan, would some of your patches that inhibit the MSI write affect this
>in a PV guest?
As far as I can tell, this patch should not cause issue to VT-d side, instead,
it''s more about access from CPU.
Can you try to pass a option to xen as "iommu=off", and check the
result?

Thanks
--jyh
>
>>
>> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary Pending Fault
>> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device [07:00.0]
>> fault addr
>c00000, iommu reg = ffff82c3fff57000
>> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
>> (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf = 7:0.0 gmfn =
c00
>> (XEN)     root_entry = ffff83019ff70000
>> (XEN)     root_entry[7] = 19cf52001
>> (XEN)     context = ffff83019cf52000
>> (XEN)     context[0] = 102_706dc005
>> (XEN)     l4 = ffff8300706dc000
>> (XEN)     l4_index = 0
>> (XEN)     l4[0] = 706db003
>> (XEN)     l3 = ffff8300706db000
>> (XEN)     l3_index = 0
>> (XEN)     l3[0] = 706da003
>> (XEN)     l2 = ffff8300706da000
>> (XEN)     l2_index = 6
>> (XEN)     l2[6] = 0
>>
>>
>> -Ray
>>
>>
>> ________________________________
>> From: xen-devel-bounces@lists.xensource.com
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Bruce Edge
>> Sent: Monday, September 27, 2010 9:46 PM
>> To: Jiang, Yunhong
>> Cc: xen-devel@lists.xensource.com; Konrad Rzeszutek Wilk
>> Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts
>> on Nehalem
>>
>> On Mon, Sep 27, 2010 at 8:26 PM, Jiang, Yunhong
><yunhong.jiang@intel.com<mailto:yunhong.jiang@intel.com>> wrote:
>> "xm dmesg" should gives xen''s boot log, and
sometimes it contain some
>> helpful
>information, I think, especially loglvl and guest_loglvl is set to all.
>>
>> I looked at the xm dmesg output and there''s nothing more than
what I
>> already
>provided, aside from a bunch of commands from me poking at it.
>>
>> -Bruce
>>
>>
>> Thanks
>> --jyh
>>
>> From: Bruce Edge
>[mailto:bruce.edge@gmail.com<mailto:bruce.edge@gmail.com>]
>> Sent: Tuesday, September 28, 2010 11:16 AM
>> To: Jiang, Yunhong
>> Cc: Konrad Rzeszutek Wilk;
>xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>
>>
>> Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts
>> on Nehalem
>>
>> On Mon, Sep 27, 2010 at 6:15 PM, Jiang, Yunhong
><yunhong.jiang@intel.com<mailto:yunhong.jiang@intel.com>> wrote:
>> Is the 07:0.0 your tachyon device? The VT-d fault is suspcious.
>>
>> Yes, there is 1 quad port card is this sytem:
>>
>> 07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
>> 07:00.1 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
>> 07:00.2 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
>> 07:00.3 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 08)
>>
>>
>> Also is it possible to share the xen output?
>>
>> I attached the dom0 boot output. Let me know if you wanted something
else.
>>
>> Also, here''s the dom0 console output upon starting the VM:
This
>> lockdep error
>started with the release of 2.6.32.21. Note that I''m running the
same
>kernel for the domU and dom0.
>>
>> [ 1817.684097] ------------[ cut here ]------------ [ 1817.684113]
>> WARNING: at kernel/lockdep.c:2323
>trace_hardirqs_on_caller+0x12f/0x190()
>> [ 1817.684119] Hardware name: ProLiant DL380 G6 [ 1817.684122]
>> Modules linked in: xt_physdev ipv6 osa_mfgdom0 xenfs xen_gntdev
>fbcon tileblit font bitblit softcursor xen_evtchn xen_pciback radeon
>ttm drm_kms_helper tun drm i2c_algo_bit ipmi_si i2c_core
>ipmi_msghandler joydev serio_raw hpwdt hpilo bridge stp llc usbhid hid
>cciss usb_storage
>> [ 1817.684190] Pid: 11, comm: xenwatch Not tainted
>> 2.6.32.21-xenoprof-1 #1 [ 1817.684195] Call Trace:
>> [ 1817.684197]  <IRQ>  [<ffffffff810aa18f>] ?
>trace_hardirqs_on_caller+0x12f/0x190
>> [ 1817.684209]  [<ffffffff8106bed0>]
warn_slowpath_common+0x80/0xd0 [
>> 1817.684217]  [<ffffffff815f2b80>] ? _spin_unlock_irq+0x30/0x40 [
>> 1817.684223]  [<ffffffff8106bf34>] warn_slowpath_null+0x14/0x20 [
>> 1817.684229]  [<ffffffff810aa18f>]
>> trace_hardirqs_on_caller+0x12f/0x190
>> [ 1817.684234]  [<ffffffff810aa1fd>] trace_hardirqs_on+0xd/0x10 [
>> 1817.684240]  [<ffffffff815f2b80>] _spin_unlock_irq+0x30/0x40 [
>> 1817.684266]  [<ffffffff813c4fc5>]
>> add_to_net_schedule_list_tail+0x85/0xd0
>> [ 1817.684271]  [<ffffffff813c6216>] netif_be_int+0x36/0x160 [
>> 1817.684278]  [<ffffffff810e10d0>] handle_IRQ_event+0x70/0x180 [
>> 1817.684284]  [<ffffffff810e36e9>] handle_edge_irq+0xc9/0x170 [
>> 1817.684291]  [<ffffffff813b8d7f>]
__xen_evtchn_do_upcall+0x1bf/0x1f0
>> [ 1817.684297]  [<ffffffff813b92fd>]
xen_evtchn_do_upcall+0x3d/0x60 [
>> 1817.684304]  [<ffffffff8101647e>]
>> xen_do_hypervisor_callback+0x1e/0x30
>> [ 1817.684308]  <EOI>  [<ffffffff8100940a>] ?
>> hypercall_page+0x40a/0x1010 [ 1817.684319]  [<ffffffff8100940a>]
?
>> hypercall_page+0x40a/0x1010 [ 1817.684325]  [<ffffffff813bce54>]
?
>> xb_write+0x1e4/0x290 [ 1817.684330]  [<ffffffff813bd8ca>] ?
>> xs_talkv+0x6a/0x1f0 [ 1817.684336]  [<ffffffff813bd8d8>] ?
>> xs_talkv+0x78/0x1f0 [ 1817.684341]  [<ffffffff813bdbcd>] ?
>> xs_single+0x4d/0x60 [ 1817.684346]  [<ffffffff813be502>] ?
>> xenbus_read+0x52/0x80 [ 1817.684352]  [<ffffffff813c87fc>] ?
>> frontend_changed+0x48c/0x770 [ 1817.684358]  [<ffffffff813bf76d>]
?
>> xenbus_otherend_changed+0xdd/0x1b0
>> [ 1817.684365]  [<ffffffff8101122f>] ?
>> xen_restore_fl_direct_end+0x0/0x1 [ 1817.684371]
>> [<ffffffff810ac830>] ? lock_release+0xb0/0x230 [ 1817.684376]
>> [<ffffffff813bfae0>] ? frontend_changed+0x10/0x20 [ 1817.684382]
>> [<ffffffff813bd4f5>] ? xenwatch_thread+0x55/0x160 [ 1817.684389]
>> [<ffffffff81093400>] ? autoremove_wake_function+0x0/0x40 [
>> 1817.684394]  [<ffffffff813bd4a0>] ? xenwatch_thread+0x0/0x160 [
>> 1817.684400]  [<ffffffff81093086>] ? kthread+0x96/0xb0 [
1817.684405]
>> [<ffffffff8101632a>] ? child_rip+0xa/0x20 [ 1817.684410]
>> [<ffffffff81015c90>] ? restore_args+0x0/0x30 [ 1817.684415]
>> [<ffffffff81016320>] ? child_rip+0x0/0x20
>>
>> -Bruce
>>
>>
>>
>> Thanks
>> --jyh
>>
>> >-----Original Message-----
>> >From:
>xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounces@lists.xe
>nsourc
>e.com>
>>
>[mailto:xen-devel-bounces@lists.xensource.com<mailto:xen-devel-bounc
>> >es@list
>s.xensource.com>] On Behalf Of Bruce Edge
>> >Sent: Tuesday, September 28, 2010 7:54 AM
>> >To: Konrad Rzeszutek Wilk
>> >Cc:
>>
>xen-devel@lists.xensource.com<mailto:xen-devel@lists.xensource.com>
>> >Subject: Re: [Xen-devel] pv-ops domU not working with MSI
interrupts
>> >on
>Nehalem
>> >
>> >On Mon, Sep 27, 2010 at 12:54 PM, Konrad Rzeszutek Wilk
>> ><konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>>
wrote:
>> >> On Mon, Sep 27, 2010 at 12:16:50PM -0700, Bruce Edge wrote:
>> >>> On Mon, Sep 27, 2010 at 10:24 AM, Konrad Rzeszutek Wilk
>> >>>
<konrad.wilk@oracle.com<mailto:konrad.wilk@oracle.com>> wrote:
>> >>> >
>> >>> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge
wrote:
>> >>> > > One of our developers who is working on a
tachyon driver is
>> >>> > > complaining that the pvops domU kernel is not
working for
>> >>> > > these MSI interrupts.
>> >>> > > This is using the current head of xen/2.6.32.x
on both a
>> >>> > > single Nahelam 920 and a dual E5540. This
behavior is
>> >>> > > consistent with Xen 4.0.1, 4.0.2.rc1-pre and
4.1.
>> >>> > >
>> >>> > > Here are his comments:
>> >>> > >
>> >>> > > - the driver has no problem to enable msi
interrupt and
>> >>> > > request the interrupt through kernel functions
pci_enable_msi
>> >>> > > & request_irq
>> >>> >
>> >>> > What shows up in the Xen console when you send the
''q'' key?
>> >>> > Does it show that the vector is assigned to the
appropiate guest?
>> >>>
>> >>> The Xen console q key shows that the domU is assigned:
>> >>>
>> >>> (XEN)     Interrupts { 32, 41-42, 47 }
>> >>
>> >> Aha!
>> >>
>> >>>
>> >>> but the domU thinks it has:
>> >>>
>> >>> 124/125/126/127
>> >>>
>> >>> Is there some mapping that''s taking place, or is
this plain wrong?
>> >>
>> >> That looks wrong. The IRQ numbers (even though they are MSI
>> >> vectors) are setup as IRQ numbers in the DomU guest. You
should
>> >> have seen
>> >>
>> >> 32:
>> >> 41:
>> >> 42:
>> >> 47:
>> >> in you /proc/interrupts on your DomU guest.
>> >>
>> >> I wonder what broke  - can you use
>>
>git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http://
>> >git.kernel.org
>/pub/scm/linux/kernel/git/konrad/xen.git>
>> >> devel/xen-pcifront-0.5 (or pv/pcifront-2.6.32)?
>> >
>> >Please forgive the git ignorance.
>> >
>> >Is this the right syntax?
>> >
>> >git clone
>git://git.kernel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.32
><http://git.ke
>rnel.org/pub/scm/linux/kernel/git/konrad:pv/pcifront-2.6.32>
>> >linux-2.6.32-pv-pcifront
>> >
>> >Initialized empty Git repository in
>>
>/import/kaan/bedge/src/xen/kernel/pv-ops/linux-2.6.32-pv-pcifront/.g
>> >it/
>> >fatal: The remote end hung up unexpectedly
>> >
>> >Or:
>> >
>> > git clone
>git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git<http://git
>.kernel.org/p ub/scm/linux/kernel/git/konrad/xen.git>
>> >
>> >Initialized empty Git repository in
>> >/import/kaan/bedge/src/xen/kernel/pv-ops/xen/.git/
>> >remote: error: Could not read
>> >59eab2f8f04147c5aadc99f2034ca7e5b81e890f
>> >remote: fatal: Failed to traverse parents of commit
>> >979e121cb348add17ed8171bf447b27a3a9d1be3
>> >remote: aborting due to possible repository corruption on the
remote side.
>> >fatal: early EOF
>> >fatal: index-pack failed
>> >
>> >>
>> >> It has the latest pcifront driver but without the PVonHVM
>> >> enhancments so we can try to eliminate the PvONHVM logic out
of the picture.
>> >>
>> >>>
>> >>> >
>> >>> > > - the interrupt does happen. But the interrupt
service
>> >>> > > routine of tachyon driver doesn''t
detect any interrupt status
>> >>> > > related to this interrupt, which inhibits the
tachyon chip
>> >>> > > from coming on-line. And there are high count of
tachyon
>> >>> > > interrupt in /proc/interrupts
>> >>> >
>> >>> > Is it checking the PCI_STATUS_INTERRUPT or the
appropiate
>> >>> > register in the MMIO BAR?
>> >>> >
>> >>>
>> >>> The driver would check the appropriate register (tachyon
>> >>> registers) in the MMIO to determine the source of
interrupts.
>> >>
>> >> OK, so that isn''t it. Is there anything at these
vectors:
>> >> 7c, 7d, 7e, and 7f? When you use xen debug-keys
''i'' or ''q'' it
>> >> should give you an inkling what device this is set for.
>> >
>> >When I run a distro kernel in hvm mode, I get the expected irq
mappings:
>> >
>> >''i'' - Note 66 - 69
>> >(XEN)    IRQ:  66 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:3a
>> >type=PCI-MSI         status=00000010 in-flight=0
>> >domain-list=10:127(----),
>> >(XEN)    IRQ:  67 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:42
>> >type=PCI-MSI         status=00000010 in-flight=0
>> >domain-list=10:126(----),
>> >(XEN)    IRQ:  68 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:4a
>> >type=PCI-MSI         status=00000010 in-flight=0
>> >domain-list=10:125(----),
>> >(XEN)    IRQ:  69 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:52
>> >type=PCI-MSI         status=00000010 in-flight=0
>> >domain-list=10:124(----)
>> >
>> >
>> >''q''
>> >(XEN)     Interrupts { 32, 41-42, 47, 124-127 }
>> >
>> >
>> >The same data with pv-ops kernel shows:
>> >
>> >''i''
>> >IRQ numbers stop at 65, no 66 - 69 present:
>> >
>> >(XEN)    IRQ:  63 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:91
>> >type=PCI-MSI         status=00000010 in-flight=0
>> >domain-list=0:289(----),
>> >(XEN)    IRQ:  64 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:99
>> >type=PCI-MSI         status=00000002 mapped, unbound
>> >(XEN)    IRQ:  65 affinity:ffffffff,ffffffff,ffffffff,ffffffff
vec:b1
>> >type=PCI-MSI         status=00000010 in-flight=0
>> >domain-list=0:287(----),
>> >(XEN) IO-APIC interrupt information:
>> >
>> >''q''
>> >(XEN)     Interrupts { 32, 41-42, 47 }
>> >
>> >>
>> >>>
>> >>> > >
>> >>> > > kaan-18-dpm:~# cat /proc/interrupts | grep TACH
>> >>> > >
>> >124:     760415          0          0          0          0
>> >    0
>> >>> > >          0          0          0          0     
0
>> >      0
>> >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
>> >>> > >
>> >125:     762234          0          0          0          0
>> >    0
>> >>> > >          0          0          0          0     
0
>> >      0
>> >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
>> >>> > >
>> >126:     764180          0          0          0          0
>> >    0
>> >>> > >          0          0          0          0     
0
>> >      0
>> >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
>> >>> > >
>> >127:     764164          0          0          0          0
>> >    0
>> >>> > >          0          0          0          0     
0
>> >      0
>> >>> > >     0          0  xen-pirq-pcifront-msi 
HW_TACHYON
>> >>> >
>> >>> > Can you provide the full dmesg output?
>> >>>
>> >>> Attached.
>> >>>
>> >>> Some possibly related messages on dom0 console:
>> >>>
>> >>> [ 1882.269778] pciback 0000:07:00.0: enabling device (0000
->
>> >>> 0003) [ 1882.269800] xen: registering gsi 32 triggering 0
>> >>> polarity 1 [ 1882.269827] xen_allocate_pirq: returning irq
32 for
>> >>> gsi 32 [ 1882.269834] xen: --> irq=32 [ 1882.269841]
Already
>> >>> setup the GSI :32 [ 1882.269847] pciback 0000:07:00.0: PCI
INT A
>> >>> -> GSI 32 (level, low) -> IRQ 32 [ 1882.269866]
pciback
>> >>> 0000:07:00.0: setting latency timer to 64 [ 1882.270463]
pciback
>> >>> 0000:07:00.0: Driver tried to write to a read-only
configuration
>> >>> space field at offset 0x62, size 2. This may be harmless,
but if
>> >>> you have problems with your device:
>> >>
>> >> Uhhh, for that I think you need to do ''lspci -vvv
-xxx -s 07:00.00''
>> >> to find out what is at the configuration space. You could
enable
>> >> it using the permissive attribute.
>> >>
>> >>> [ 1882.270465] 1) see permissive attribute in sysfs [
>> >>> 1882.270467] 2) report problems to the xen-devel mailing
list
>> >>> along with details of your device obtained from lspci.
>> >>> [ 1882.270615]   alloc irq_desc for 478 on node 0
>> >>> [ 1882.270625]   alloc kstat_irqs on node 0
>> >>
>> >> So for 478: what do you see? xen-pciback I presume?
>> >>> [ 1882.348411] pciback 0000:07:00.1: enabling device (0000
->
>> >>> 0003) [ 1882.348433] xen: registering gsi 42 triggering 0
>> >>> polarity 1 [ 1882.348440] xen_allocate_pirq: returning irq
42 for
>> >>> gsi 42 [ 1882.348445] xen: --> irq=42 [ 1882.348472]
Already
>> >>> setup the GSI :42 [ 1882.348479] pciback 0000:07:00.1: PCI
INT B
>> >>> -> GSI 42 (level, low) -> IRQ 42 [ 1882.348497]
pciback
>> >>> 0000:07:00.1: setting latency timer to 64 [ 1882.349063]
pciback
>> >>> 0000:07:00.1: Driver tried to write to a read-only
configuration
>> >>> space field at offset 0x62, size 2. This may be harmless,
but if
>> >>> you have problems with your device:
>> >>> [ 1882.349066] 1) see permissive attribute in sysfs [
>> >>> 1882.349067] 2) report problems to the xen-devel mailing
list
>> >>> along with details of your device obtained from lspci.
>> >>> [ 1882.349205]   alloc irq_desc for 477 on node 0
>> >>> [ 1882.349215]   alloc kstat_irqs on node 0
>> >>> [ 1882.402893] pciback 0000:07:00.2: enabling device (0000
->
>> >>> 0003) [ 1882.402908] xen: registering gsi 47 triggering 0
>> >>> polarity 1 [ 1882.402913] xen_allocate_pirq: returning irq
47 for
>> >>> gsi 47 [ 1882.402916] xen: --> irq=47 [ 1882.402921]
Already
>> >>> setup the GSI :47 [ 1882.402925] pciback 0000:07:00.2: PCI
INT C
>> >>> -> GSI 47 (level, low) -> IRQ 47 [ 1882.402938]
pciback
>> >>> 0000:07:00.2: setting latency timer to 64 [ 1882.403280]
pciback
>> >>> 0000:07:00.2: Driver tried to write to a read-only
configuration
>> >>> space field at offset 0x62, size 2. This may be harmless,
but if
>> >>> you have problems with your device:
>> >>> [ 1882.403282] 1) see permissive attribute in sysfs [
>> >>> 1882.403282] 2) report problems to the xen-devel mailing
list
>> >>> along with details of your device obtained from lspci.
>> >>> [ 1882.403380]   alloc irq_desc for 476 on node 0
>> >>> [ 1882.403386]   alloc kstat_irqs on node 0
>> >>> (XEN) [VT-D]iommu.c:824: iommu_fault_status: Primary
Pending
>> >>> Fault
>> >>> (XEN) [VT-D]iommu.c:799: DMAR:[DMA Write] Request device
>> >>> [07:00.0] fault addr e6f80000, iommu reg =
ffff82c3fff57000
>> >>> (XEN) DMAR:[fault reason 05h] PTE Write access is not set
>> >>> (XEN) print_vtd_entries: iommu = ffff83019fffa370 bdf =
7:0.0 gmfn = e6f80
>> >>> (XEN)     root_entry = ffff83019ff70000
>> >>> (XEN)     root_entry[7] = 19cf52001
>> >>> (XEN)     context = ffff83019cf52000
>> >>> (XEN)     context[0] = 102_706dc005
>> >>> (XEN)     l4 = ffff8300706dc000
>> >>> (XEN)     l4_index = 0
>> >>> (XEN)     l4[0] = 706db003
>> >>> (XEN)     l3 = ffff8300706db000
>> >>> (XEN)     l3_index = 3
>> >>> (XEN)     l3[3] = 702b6003
>> >>> (XEN)     l2 = ffff8300702b6000
>> >>> (XEN)     l2_index = 137
>> >>> (XEN)     l2[137] = 0
>> >>> (XEN)     l2[137] not present
>> >>> (XEN) traps.c:466:d0 Unhandled nmi fault/trap [#2] on VCPU
0
>> >>> [ec=0000]
>> >>
>> >> That is not good. What changed from your earlier emails that
this
>> >> was
>triggered?
>> >
>> >Nothing
>> >> Or was it triggered all along?
>> >
>> >Yes, I just included it for completeness
>> >
>> >> What happens if you run the system without the iommu enabled?
>> >
>> >Haven''t tried yet. Will check that next.
>> >
>> >-Bruce
>> >
>> >_______________________________________________
>> >Xen-devel mailing list
>>
>Xen-devel@lists.xensource.com<mailto:Xen-devel@lists.xensource.com>
>> >http://lists.xensource.com/xen-devel
>>
>>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Bruce Edge

2010-Sep-30 18:55 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Tue, Sep 28, 2010 at 7:56 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:>>  git clone
 git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git
>>
>> Initialized empty Git repository in
>> /import/kaan/bedge/src/xen/kernel/pv-ops/xen/.git/
>> remote: error: Could not read 59eab2f8f04147c5aadc99f2034ca7e5b81e890f
>> remote: fatal: Failed to traverse parents of commit
>> 979e121cb348add17ed8171bf447b27a3a9d1be3
>> remote: aborting due to possible repository corruption on the remote
side.
>> fatal: early EOF
>> fatal: index-pack failed
>
> That should have worked, but it looks as my git repo is busted. Let me fix
that
> and once it done you should be able to do
>
> cd xen
> git checkout origin/pv/pcifront-2.6.32
That''s working now, will test today.
>
>>
>> >
>> > It has the latest pcifront driver but without the PVonHVM
enhancments
>> > so we can try to eliminate the PvONHVM logic out of the picture.
>> >
>> >>
>> >> >
>> >> > > - the interrupt does happen. But the interrupt
service routine of
>> >> > > tachyon driver doesn''t detect any interrupt
status related to this
>> >> > > interrupt, which inhibits the tachyon chip from
coming on-line. And
>> >> > > there are high count of tachyon interrupt in
/proc/interrupts
>> >> >
>> >> > Is it checking the PCI_STATUS_INTERRUPT or the appropiate
register
>> >> > in the MMIO BAR?
>> >> >
>> >>
>> >> The driver would check the appropriate register (tachyon
registers) in
>> >> the MMIO to determine the source of interrupts.
>> >
>> > OK, so that isn''t it. Is there anything at these vectors:
>> > 7c, 7d, 7e, and 7f? When you use xen debug-keys
''i'' or ''q'' it should give you
>> > an inkling what device this is set for.
>>
>> When I run a distro kernel in hvm mode, I get the expected irq
mappings:
>>
>> ''i'' - Note 66 - 69
>> (XEN)    IRQ:  66 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:3a
>> type=PCI-MSI         status=00000010 in-flight=0
>> domain-list=10:127(----),
>> (XEN)    IRQ:  67 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:42
>> type=PCI-MSI         status=00000010 in-flight=0
>> domain-list=10:126(----),
>> (XEN)    IRQ:  68 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:4a
>> type=PCI-MSI         status=00000010 in-flight=0
>> domain-list=10:125(----),
>> (XEN)    IRQ:  69 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:52
>> type=PCI-MSI         status=00000010 in-flight=0
>> domain-list=10:124(----)
>>
>
> What does cat /proc/interrupts (don''t do the grep) for this HVM
guest?
Here''s what''s reported for the domU in hvm mode:

~ # cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5
  0:         28          0          0          0          0          0
  IO-APIC-edge      timer
  1:         18          0          0          0          0          0
  IO-APIC-edge      i8042
  6:          2          0          0          0          0          0
  IO-APIC-edge      floppy
  7:          0          0          0          0          0          0
  IO-APIC-edge      parport0
  8:          0          0          0          0          0          0
  IO-APIC-edge      rtc0
  9:          0          0          0          0          0          0
  IO-APIC-fasteoi   acpi
 12:          2          0          0          0          0          0
  IO-APIC-edge      i8042
 14:      67771          0          0          0          0          0
  IO-APIC-edge      ata_piix
 15:       3044          0          0          0          0          0
  IO-APIC-edge      ata_piix
 23:         22          0          0          0          0          0
  IO-APIC-fasteoi   uhci_hcd:usb1
 32:        953          0          0          0          0          0
  IO-APIC-fasteoi   eth0
NMI:          0          0          0          0          0          0
  Non-maskable interrupts
LOC:      26320      15814      13000      13825      11125      10458
  Local timer interrupts
SPU:          0          0          0          0          0          0
  Spurious interrupts
PMI:          0          0          0          0          0          0
  Performance monitoring interrupts
PND:          0          0          0          0          0          0
  Performance pending work
RES:      78730      21174      25236      34071      25122      26924
  Rescheduling interrupts
CAL:         90        806        808        488        587        392
  Function call interrupts
TLB:       3429       2358       3475       2522       3986       2684
  TLB shootdowns
TRM:          0          0          0          0          0          0
  Thermal event interrupts
THR:          0          0          0          0          0          0
  Threshold APIC interrupts
MCE:          0          0          0          0          0          0
  Machine check exceptions
MCP:          2          2          2          2          2          2
  Machine check polls
ERR:          0
MIS:          0

~ # lspci

00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB Controller: Intel Corporation 82371SB PIIX3 USB
[Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 Class ff80: XenSource, Inc. Xen Platform Device (rev 01)
00:04.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL-8139/8139C/8139C+ (rev 20)
00:10.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 05)
00:11.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 05)
00:12.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 05)
00:13.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 05)
>>
>> ''q''
>> (XEN)     Interrupts { 32, 41-42, 47, 124-127 }
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk

2010-Oct-01 21:11 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge
wrote:> One of our developers who is working on a tachyon driver is
> complaining that the pvops domU kernel is not working for these MSI
> interrupts.
> This is using the current head of xen/2.6.32.x on both a single
> Nahelam 920 and a dual E5540. This behavior is consistent with Xen
> 4.0.1, 4.0.2.rc1-pre and 4.1.

I just checked on my SuperMicro X8DTN, this combination
 - For Dom0, git commit fe999249 (2.6.32.18)
 - For DomU, devel/xen-pcifront-0.6 or devel/xen-pcifront-0.7
 - For Hypervisor I used cs 21976, but found that the latest (22155) works too

with which where I passed in PCI devices with legacy IRQ, MSI, and MSI-X. I
tried
a combination of doing this with IOMMU (VT-d) and without - both cases these
devices:

00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #3 (rev 02)
00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI
Controller #1 (rev 02)
03:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet
Controller (Copper) (rev 06)
0a:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network
Connection (rev 02)

worked just fine (either defining pci=["..."] or just using
pci-attach).
 
But if I use the latest xen/next or xen/stable-2.6.32.x it does not look
that happy :-(


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Bruce Edge

2010-Oct-01 21:19 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Fri, Oct 1, 2010 at 2:11 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:> On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge wrote:
>> One of our developers who is working on a tachyon driver is
>> complaining that the pvops domU kernel is not working for these MSI
>> interrupts.
>> This is using the current head of xen/2.6.32.x on both a single
>> Nahelam 920 and a dual E5540. This behavior is consistent with Xen
>> 4.0.1, 4.0.2.rc1-pre and 4.1.
>
>
> I just checked on my SuperMicro X8DTN, this combination
>  - For Dom0, git commit fe999249 (2.6.32.18)
>  - For DomU, devel/xen-pcifront-0.6 or devel/xen-pcifront-0.7
>  - For Hypervisor I used cs 21976, but found that the latest (22155) works
too
Perfect! That''s all I need - a recipe that works. Thank you very much.

-Bruce
>
> with which where I passed in PCI devices with legacy IRQ, MSI, and MSI-X. I
tried
> a combination of doing this with IOMMU (VT-d) and without - both cases
these devices:
>
> 00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #1 (rev 02)
> 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #2 (rev 02)
> 00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #3 (rev 02)
> 00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI
Controller #1 (rev 02)
> 03:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet
Controller (Copper) (rev 06)
> 0a:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network
Connection (rev 02)
>
> worked just fine (either defining pci=["..."] or just using
pci-attach).
>
> But if I use the latest xen/next or xen/stable-2.6.32.x it does not look
> that happy :-(
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Bruce Edge

2010-Oct-01 23:30 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Fri, Oct 1, 2010 at 2:11 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:> On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge wrote:
>> One of our developers who is working on a tachyon driver is
>> complaining that the pvops domU kernel is not working for these MSI
>> interrupts.
>> This is using the current head of xen/2.6.32.x on both a single
>> Nahelam 920 and a dual E5540. This behavior is consistent with Xen
>> 4.0.1, 4.0.2.rc1-pre and 4.1.
>
>
> I just checked on my SuperMicro X8DTN, this combination
>  - For Dom0, git commit fe999249 (2.6.32.18)
>  - For DomU, devel/xen-pcifront-0.6 or devel/xen-pcifront-0.7
Konrad,

Apologies in advance for the rookie git questions.

I''m assuming the git repos you mentioned are based on
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git ?

Your devel/xen-pcifront-0.7 isn''t visible:

%> git branch -r  | grep devel/xen-pcifront

  origin/devel/xen-pcifront-0.1
  origin/devel/xen-pcifront-0.2
  origin/devel/xen-pcifront-0.3
  origin/devel/xen-pcifront-0.4
  origin/devel/xen-pcifront-0.5
  origin/devel/xen-pcifront-0.6


and your devel/xen-pcifront-0.6 forces a config restart:

0 %> make oldconfig
scripts/kconfig/conf --oldconfig arch/x86/Kconfig
*
* Restart config...
*
*
* General setup
*
Prompt for development and/or incomplete code/drivers (EXPERIMENTAL) [Y/n/?] y
Cross-compiler tool prefix (CROSS_COMPILE) [] (NEW) ^Cmake[1]: ***
[oldconfig] Interrupt
make: *** [oldconfig] Interrupt

Is there a trick to avoiding the config restart?

-Bruce
>  - For Hypervisor I used cs 21976, but found that the latest (22155) works
too
>
> with which where I passed in PCI devices with legacy IRQ, MSI, and MSI-X. I
tried
> a combination of doing this with IOMMU (VT-d) and without - both cases
these devices:
>
> 00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #1 (rev 02)
> 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #2 (rev 02)
> 00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #3 (rev 02)
> 00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI
Controller #1 (rev 02)
> 03:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet
Controller (Copper) (rev 06)
> 0a:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network
Connection (rev 02)
>
> worked just fine (either defining pci=["..."] or just using
pci-attach).
>
> But if I use the latest xen/next or xen/stable-2.6.32.x it does not look
> that happy :-(
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Bruce Edge

2010-Oct-03 05:46 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Fri, Oct 1, 2010 at 4:30 PM, Bruce Edge <bruce.edge@gmail.com>
wrote:> On Fri, Oct 1, 2010 at 2:11 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
>> On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge wrote:
>>> One of our developers who is working on a tachyon driver is
>>> complaining that the pvops domU kernel is not working for these MSI
>>> interrupts.
>>> This is using the current head of xen/2.6.32.x on both a single
>>> Nahelam 920 and a dual E5540. This behavior is consistent with Xen
>>> 4.0.1, 4.0.2.rc1-pre and 4.1.
>>
>>
>> I just checked on my SuperMicro X8DTN, this combination
>>  - For Dom0, git commit fe999249 (2.6.32.18)
>>  - For DomU, devel/xen-pcifront-0.6 or devel/xen-pcifront-0.7
>
> Konrad,
>
> Apologies in advance for the rookie git questions.
>
> I''m assuming the git repos you mentioned are based on
> git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git ?
>
> Your devel/xen-pcifront-0.7 isn''t visible:
>
> %> git branch -r  | grep devel/xen-pcifront
>
>  origin/devel/xen-pcifront-0.1
>  origin/devel/xen-pcifront-0.2
>  origin/devel/xen-pcifront-0.3
>  origin/devel/xen-pcifront-0.4
>  origin/devel/xen-pcifront-0.5
>  origin/devel/xen-pcifront-0.6
>
>
> and your devel/xen-pcifront-0.6 forces a config restart:
>
> 0 %> make oldconfig
> scripts/kconfig/conf --oldconfig arch/x86/Kconfig
> *
> * Restart config...
> *
> *
> * General setup
> *
> Prompt for development and/or incomplete code/drivers (EXPERIMENTAL)
[Y/n/?] y
> Cross-compiler tool prefix (CROSS_COMPILE) [] (NEW) ^Cmake[1]: ***
> [oldconfig] Interrupt
> make: *** [oldconfig] Interrupt
>
> Is there a trick to avoiding the config restart?
Found the problem here - I was using a .config from a 2.6.32 kernel.
There''s something in the 2.6.36 config parsing that declares it
invalid and starts over.
Starting with a clean ''make defconfig'' and editing that works
fine.

I had never see this behavior in kernel builds before. I thought the
.config files were always compatible between versions.

-Bruce
>
> -Bruce
>
>>  - For Hypervisor I used cs 21976, but found that the latest (22155)
works too
>>
>> with which where I passed in PCI devices with legacy IRQ, MSI, and
MSI-X. I tried
>> a combination of doing this with IOMMU (VT-d) and without - both cases
these devices:
>>
>> 00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #1 (rev 02)
>> 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #2 (rev 02)
>> 00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #3 (rev 02)
>> 00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2
EHCI Controller #1 (rev 02)
>> 03:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet
Controller (Copper) (rev 06)
>> 0a:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network
Connection (rev 02)
>>
>> worked just fine (either defining pci=["..."] or just using
pci-attach).
>>
>> But if I use the latest xen/next or xen/stable-2.6.32.x it does not
look
>> that happy :-(
>>
>>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Pasi Kärkkäinen

2010-Oct-03 12:08 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Sat, Oct 02, 2010 at 10:46:45PM -0700, Bruce Edge
wrote:> >
> > and your devel/xen-pcifront-0.6 forces a config restart:
> >
> > 0 %> make oldconfig
> > scripts/kconfig/conf --oldconfig arch/x86/Kconfig
> > *
> > * Restart config...
> > *
> > *
> > * General setup
> > *
> > Prompt for development and/or incomplete code/drivers (EXPERIMENTAL)
[Y/n/?] y
> > Cross-compiler tool prefix (CROSS_COMPILE) [] (NEW) ^Cmake[1]: ***
> > [oldconfig] Interrupt
> > make: *** [oldconfig] Interrupt
> >
> > Is there a trick to avoiding the config restart?
> 
> Found the problem here - I was using a .config from a 2.6.32 kernel.
> There''s something in the 2.6.36 config parsing that declares it
> invalid and starts over.
> Starting with a clean ''make defconfig'' and editing that
works fine.
> 
> I had never see this behavior in kernel builds before. I thought the
> .config files were always compatible between versions.
> 
You need to do "make oldconfig" to adapt/merge the 2.6.32 config to
2.6.36.

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk

2010-Oct-04 14:48 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

> git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git ?
> 
> Your devel/xen-pcifront-0.7 isn''t visible:
It should be now. I pushed it late on Friday and repushed it today since it had
some commits with the wrong committer field.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Bruce Edge

2010-Oct-05 18:21 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Mon, Oct 4, 2010 at 7:48 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:>> git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git ?
>>
>> Your devel/xen-pcifront-0.7 isn''t visible:
>
> It should be now. I pushed it late on Friday and repushed it today since it
had
> some commits with the wrong committer field.
>
Is there anything special one needs to do when building this kernel?

The bzImage target err''s out with undef''d syms:

  CC      init/version.o
  LD      init/built-in.o
  LD      .tmp_vmlinux1
drivers/built-in.o: In function `notify_remote_via_evtchn'':
xen-pcifront.c:(.text+0x15b79): undefined reference to `hypercall_page''
xen-pcifront.c:(.text+0x15b9d): undefined reference to `hypercall_page''
drivers/built-in.o: In function `do_pci_op'':
xen-pcifront.c:(.text+0x15cce): undefined reference to
`xen_clear_irq_pending''
xen-pcifront.c:(.text+0x15ce6): undefined reference to
`xen_poll_irq_timeout''
xen-pcifront.c:(.text+0x15cee): undefined reference to
`xen_clear_irq_pending''
drivers/built-in.o: In function `pcifront_xenbus_probe'':
xen-pcifront.c:(.text+0x161db): undefined reference to `xen_features''
xen-pcifront.c:(.text+0x161fc): undefined reference to
`xenbus_dev_fatal''
xen-pcifront.c:(.text+0x16209): undefined reference to
`get_phys_to_machine''
xen-pcifront.c:(.text+0x16221): undefined reference to
`xenbus_grant_ring''
xen-pcifront.c:(.text+0x1623b): undefined reference to
`xenbus_alloc_evtchn''
xen-pcifront.c:(.text+0x16261): undefined reference to
`bind_evtchn_to_irqhandler''
xen-pcifront.c:(.text+0x16273): undefined reference to
`xenbus_free_evtchn''
xen-pcifront.c:(.text+0x1628b): undefined reference to
`xenbus_transaction_start''
xen-pcifront.c:(.text+0x162a6): undefined reference to
`xenbus_dev_fatal''
xen-pcifront.c:(.text+0x162ce): undefined reference to `xenbus_printf''
xen-pcifront.c:(.text+0x162f8): undefined reference to `xenbus_printf''
xen-pcifront.c:(.text+0x1631e): undefined reference to `xenbus_printf''
xen-pcifront.c:(.text+0x16332): undefined reference to
`xenbus_transaction_end''
xen-pcifront.c:(.text+0x16348): undefined reference to
`xenbus_transaction_end''
xen-pcifront.c:(.text+0x16370): undefined reference to
`xenbus_switch_state''
drivers/built-in.o: In function `pcifront_xenbus_remove'':
xen-pcifront.c:(.text+0x16861): undefined reference to
`unbind_from_irqhandler''
xen-pcifront.c:(.text+0x16871): undefined reference to
`xenbus_free_evtchn''
xen-pcifront.c:(.text+0x16883): undefined reference to
`gnttab_end_foreign_access''
drivers/built-in.o: In function `pcifront_backend_changed'':
xen-pcifront.c:(.ref.text+0xcca): undefined reference to
`xenbus_read_driver_state''
xen-pcifront.c:(.ref.text+0xd0c): undefined reference to `xenbus_scanf''
xen-pcifront.c:(.ref.text+0xd24): undefined reference to
`xenbus_dev_error''
xen-pcifront.c:(.ref.text+0xd6e): undefined reference to
`xenbus_dev_fatal''
xen-pcifront.c:(.ref.text+0xdb5): undefined reference to `xenbus_scanf''
xen-pcifront.c:(.ref.text+0xe07): undefined reference to
`xenbus_read_driver_state''
xen-pcifront.c:(.ref.text+0xe38): undefined reference to
`xenbus_read_driver_state''
xen-pcifront.c:(.ref.text+0xe63): undefined reference to `xenbus_scanf''
xen-pcifront.c:(.ref.text+0xed7): undefined reference to `xenbus_scanf''
xen-pcifront.c:(.ref.text+0xf39): undefined reference to `xenbus_scanf''
xen-pcifront.c:(.ref.text+0xfc9): undefined reference to
`xenbus_read_driver_state''
xen-pcifront.c:(.ref.text+0xff4): undefined reference to `xenbus_scanf''
xen-pcifront.c:(.ref.text+0x100c): undefined reference to
`xenbus_dev_error''
xen-pcifront.c:(.ref.text+0x1083): undefined reference to
`xenbus_scanf''
xen-pcifront.c:(.ref.text+0x10a8): undefined reference to
`xenbus_dev_fatal''
xen-pcifront.c:(.ref.text+0x10d6): undefined reference to
`xenbus_dev_fatal''
xen-pcifront.c:(.ref.text+0x10f2): undefined reference to
`xenbus_switch_state''
drivers/built-in.o: In function `pcifront_cleanup'':
xen-pcifront.c:(.exit.text+0x36): undefined reference to
`xenbus_unregister_driver''
make: *** [.tmp_vmlinux1] Error 1

I noticed the defconfig target had a couple of warnings too:

%> make defconfig

*** Default configuration is based on ''x86_64_defconfig''
warning: (XEN_PCIDEV_FRONTEND && X86) selects PCI_XEN which has unmet
direct dependencies (XEN)
warning: (XEN_PCIDEV_FRONTEND && X86) selects PCI_XEN which has unmet
direct dependencies (XEN)
#
# configuration written to .config
#

-Bruce

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk

2010-Oct-06 14:21 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Tue, Oct 05, 2010 at 11:21:38AM -0700, Bruce Edge
wrote:> On Mon, Oct 4, 2010 at 7:48 AM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> >> git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git ?
> >>
> >> Your devel/xen-pcifront-0.7 isn''t visible:
> >
> > It should be now. I pushed it late on Friday and repushed it today
since it had
> > some commits with the wrong committer field.
> >
> 
> Is there anything special one needs to do when building this kernel?
Wow. I''ve never seen those errors before. Can you send me your .config
file
so I can make sure this does not happend

For me, I''ve these options defined:

[konrad@tst002 bootstrap]$ cat linux-build/.config|grep XEN
CONFIG_XEN=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=8
CONFIG_XEN_SAVE_RESTORE=y
CONFIG_XEN_DEBUG_FS=y
CONFIG_PCI_XEN=y
CONFIG_XEN_PCIDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_FRONTEND=m
# CONFIG_NETXEN_NIC is not set
CONFIG_XEN_NETDEV_FRONTEND=m
CONFIG_XEN_KBDDEV_FRONTEND=m
CONFIG_HVC_XEN=y
CONFIG_XEN_FBDEV_FRONTEND=m
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=m
CONFIG_XENFS=m
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_PLATFORM_PCI=m
CONFIG_SWIOTLB_XEN=y


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Bruce Edge

2010-Oct-06 18:21 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Wed, Oct 6, 2010 at 7:21 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:> On Tue, Oct 05, 2010 at 11:21:38AM -0700, Bruce Edge wrote:
>> On Mon, Oct 4, 2010 at 7:48 AM, Konrad Rzeszutek Wilk
>> <konrad.wilk@oracle.com> wrote:
>> >> git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git ?
>> >>
>> >> Your devel/xen-pcifront-0.7 isn''t visible:
>> >
>> > It should be now. I pushed it late on Friday and repushed it today
since it had
>> > some commits with the wrong committer field.
>> >
>>
>> Is there anything special one needs to do when building this kernel?
>
> Wow. I''ve never seen those errors before. Can you send me your
.config file
> so I can make sure this does not happend
Those errors were from after a ''make defconfig''. I assumed my
config
was broken and that I should try the default .config to confirm that
that built before digging thought mine.

Adding the config entries below to the defconfig output does result in
a buildable kernel.

Testing out my PCI drivers with your recommended config now. Thanks.

-Bruce
>
> For me, I''ve these options defined:
>
> [konrad@tst002 bootstrap]$ cat linux-build/.config|grep XEN
> CONFIG_XEN=y
> CONFIG_XEN_PVHVM=y
> CONFIG_XEN_MAX_DOMAIN_MEMORY=8
> CONFIG_XEN_SAVE_RESTORE=y
> CONFIG_XEN_DEBUG_FS=y
> CONFIG_PCI_XEN=y
> CONFIG_XEN_PCIDEV_FRONTEND=y
> CONFIG_XEN_BLKDEV_FRONTEND=m
> # CONFIG_NETXEN_NIC is not set
> CONFIG_XEN_NETDEV_FRONTEND=m
> CONFIG_XEN_KBDDEV_FRONTEND=m
> CONFIG_HVC_XEN=y
> CONFIG_XEN_FBDEV_FRONTEND=m
> CONFIG_XEN_BALLOON=y
> CONFIG_XEN_SCRUB_PAGES=y
> CONFIG_XEN_DEV_EVTCHN=m
> CONFIG_XENFS=m
> CONFIG_XEN_COMPAT_XENFS=y
> CONFIG_XEN_SYS_HYPERVISOR=y
> CONFIG_XEN_PLATFORM_PCI=m
> CONFIG_SWIOTLB_XEN=y
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Lin, Ray

2010-Oct-08 16:48 UTC

head link

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

I just tried Bruce''s latest kernel build based on Konrad''s
devel/xen-pcifront-0.7. It doesn''t help the issue we have. The driver
still doesn''t recognize the source of interrupt, even though the
interrupts happen.


124:      87792          0          0          0          0          0     
12208          0          0          0          0          0          0         
0  xen-pirq-pcifront-msi  HW_TACHYON
125:      89692          0          0          0      10308          0         
0          0          0          0          0          0          0          0 
xen-pirq-pcifront-msi  HW_TACHYON
126:      90979          0       9021          0          0          0         
0          0          0          0          0          0          0          0 
xen-pirq-pcifront-msi  HW_TACHYON
127:     100000          0          0          0          0          0         
0          0          0          0          0          0          0          0 
xen-pirq-pcifront-msi  HW_TACHYON


-Ray

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Bruce Edge
Sent: Wednesday, October 06, 2010 11:21 AM
To: Konrad Rzeszutek Wilk
Cc: xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Wed, Oct 6, 2010 at 7:21 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:> On Tue, Oct 05, 2010 at 11:21:38AM -0700, Bruce Edge wrote:
>> On Mon, Oct 4, 2010 at 7:48 AM, Konrad Rzeszutek Wilk 
>> <konrad.wilk@oracle.com> wrote:
>> >> git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git ?
>> >>
>> >> Your devel/xen-pcifront-0.7 isn''t visible:
>> >
>> > It should be now. I pushed it late on Friday and repushed it today
>> > since it had some commits with the wrong committer field.
>> >
>>
>> Is there anything special one needs to do when building this kernel?
>
> Wow. I''ve never seen those errors before. Can you send me your
.config
> file so I can make sure this does not happend
Those errors were from after a ''make defconfig''. I assumed my
config was broken and that I should try the default .config to confirm that that
built before digging thought mine.

Adding the config entries below to the defconfig output does result in a
buildable kernel.

Testing out my PCI drivers with your recommended config now. Thanks.

-Bruce
>
> For me, I''ve these options defined:
>
> [konrad@tst002 bootstrap]$ cat linux-build/.config|grep XEN 
> CONFIG_XEN=y CONFIG_XEN_PVHVM=y
> CONFIG_XEN_MAX_DOMAIN_MEMORY=8
> CONFIG_XEN_SAVE_RESTORE=y
> CONFIG_XEN_DEBUG_FS=y
> CONFIG_PCI_XEN=y
> CONFIG_XEN_PCIDEV_FRONTEND=y
> CONFIG_XEN_BLKDEV_FRONTEND=m
> # CONFIG_NETXEN_NIC is not set
> CONFIG_XEN_NETDEV_FRONTEND=m
> CONFIG_XEN_KBDDEV_FRONTEND=m
> CONFIG_HVC_XEN=y
> CONFIG_XEN_FBDEV_FRONTEND=m
> CONFIG_XEN_BALLOON=y
> CONFIG_XEN_SCRUB_PAGES=y
> CONFIG_XEN_DEV_EVTCHN=m
> CONFIG_XENFS=m
> CONFIG_XEN_COMPAT_XENFS=y
> CONFIG_XEN_SYS_HYPERVISOR=y
> CONFIG_XEN_PLATFORM_PCI=m
> CONFIG_SWIOTLB_XEN=y
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk

2010-Oct-08 17:30 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Fri, Oct 08, 2010 at 10:48:01AM -0600, Lin, Ray
wrote:> 
> I just tried Bruce''s latest kernel build based on
Konrad''s devel/xen-pcifront-0.7. It doesn''t help the issue we
have. The driver still doesn''t recognize the source of interrupt, even
though the interrupts happen.
> 
> 
> 124:      87792          0          0          0          0          0     
12208          0          0          0          0          0          0         
0  xen-pirq-pcifront-msi  HW_TACHYON
> 125:      89692          0          0          0      10308          0     
0          0          0          0          0          0          0          0 
xen-pirq-pcifront-msi  HW_TACHYON
> 126:      90979          0       9021          0          0          0     
0          0          0          0          0          0          0          0 
xen-pirq-pcifront-msi  HW_TACHYON
> 127:     100000          0          0          0          0          0     
0          0          0          0          0          0          0          0 
xen-pirq-pcifront-msi  HW_TACHYON
> 
And you still get on the Xen hypervisor side the DMAR failure of reading the
memory?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Lin, Ray

2010-Oct-08 17:40 UTC

head link

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

I got this from dom0 when I brought up domU. There is no complaint from iommu.


-Ray

 about to get started...
(XEN) traps.c:2310:d2 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d2 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d2 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d2 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d2 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d2 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d2 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d2 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d2 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d2 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d2 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d2 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d2 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d2 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
[ 5171.932037] vif2.0: no IPv6 routers present
[ 5178.026355] blkback: ring-ref 8, event-channel 87, protocol 1 (x86_64-abi)
[ 5221.204637] pciback 0000:07:00.0: enabling device (0000 -> 0003)
[ 5221.204696] xen: registering gsi 32 triggering 0 polarity 1
[ 5221.204716] xen_allocate_pirq: returning irq 32 for gsi 32
[ 5221.204735] xen: --> irq=32
[ 5221.204749] Already setup the GSI :32
[ 5221.204764] pciback 0000:07:00.0: PCI INT A -> GSI 32 (level, low) ->
IRQ 32
[ 5221.204819] pciback 0000:07:00.0: setting latency timer to 64
[ 5221.205376]   alloc irq_desc for 474 on node 0
[ 5221.205400]   alloc kstat_irqs on node 0
[ 5221.270496] pciback 0000:07:00.1: enabling device (0000 -> 0003)
[ 5221.270536] xen: registering gsi 42 triggering 0 polarity 1
[ 5221.270576] xen_allocate_pirq: returning irq 42 for gsi 42
[ 5221.270595] xen: --> irq=42
[ 5221.270608] Already setup the GSI :42
[ 5221.270624] pciback 0000:07:00.1: PCI INT B -> GSI 42 (level, low) ->
IRQ 42
[ 5221.270660] pciback 0000:07:00.1: setting latency timer to 64
[ 5221.271210]   alloc irq_desc for 473 on node 0
[ 5221.271234]   alloc kstat_irqs on node 0
[ 5221.333809] pciback 0000:07:00.2: enabling device (0000 -> 0003)
[ 5221.333849] xen: registering gsi 47 triggering 0 polarity 1
[ 5221.333888] xen_allocate_pirq: returning irq 47 for gsi 47
[ 5221.333907] xen: --> irq=47
[ 5221.333921] Already setup the GSI :47
[ 5221.333936] pciback 0000:07:00.2: PCI INT C -> GSI 47 (level, low) ->
IRQ 47
[ 5221.333972] pciback 0000:07:00.2: setting latency timer to 64
[ 5221.334523]   alloc irq_desc for 472 on node 0
[ 5221.334546]   alloc kstat_irqs on node 0
[ 5221.595255] pciback 0000:07:00.3: enabling device (0000 -> 0003)
[ 5221.595340] xen: registering gsi 41 triggering 0 polarity 1
[ 5221.595373] xen_allocate_pirq: returning irq 41 for gsi 41
[ 5221.595422] xen: --> irq=41
[ 5221.595445] Already setup the GSI :41
[ 5221.595474] pciback 0000:07:00.3: PCI INT D -> GSI 41 (level, low) ->
IRQ 41
[ 5221.595530] pciback 0000:07:00.3: setting latency timer to 64
[ 5221.596417]   alloc irq_desc for 471 on node 0
[ 5221.596457]   alloc kstat_irqs on node 0


-----Original Message-----
From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] 
Sent: Friday, October 08, 2010 10:31 AM
To: Lin, Ray
Cc: Bruce Edge; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Fri, Oct 08, 2010 at 10:48:01AM -0600, Lin, Ray
wrote:> 
> I just tried Bruce''s latest kernel build based on
Konrad''s devel/xen-pcifront-0.7. It doesn''t help the issue we
have. The driver still doesn''t recognize the source of interrupt, even
though the interrupts happen.
> 
> 
> 124:      87792          0          0          0          0          0     
12208          0          0          0          0          0          0         
0  xen-pirq-pcifront-msi  HW_TACHYON
> 125:      89692          0          0          0      10308          0     
0          0          0          0          0          0          0          0 
xen-pirq-pcifront-msi  HW_TACHYON
> 126:      90979          0       9021          0          0          0     
0          0          0          0          0          0          0          0 
xen-pirq-pcifront-msi  HW_TACHYON
> 127:     100000          0          0          0          0          0     
0          0          0          0          0          0          0          0 
xen-pirq-pcifront-msi  HW_TACHYON
> 
And you still get on the Xen hypervisor side the DMAR failure of reading the
memory?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Bruce Edge

2010-Oct-08 17:52 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Fri, Oct 8, 2010 at 10:30 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:> On Fri, Oct 08, 2010 at 10:48:01AM -0600, Lin, Ray wrote:
>>
>> I just tried Bruce's latest kernel build based on Konrad's
devel/xen-pcifront-0.7. It doesn't help the issue we have. The driver still
doesn't recognize the source of interrupt, even though the interrupts
happen.
>>
>>
>> 124:      87792          0          0          0          0          0
     12208          0          0          0          0          0          0    
     0  xen-pirq-pcifront-msi  HW_TACHYON
>> 125:      89692          0          0          0      10308          0
         0          0          0          0          0          0          0    
     0  xen-pirq-pcifront-msi  HW_TACHYON
>> 126:      90979          0       9021          0          0          0
         0          0          0          0          0          0          0    
     0  xen-pirq-pcifront-msi  HW_TACHYON
>> 127:     100000          0          0          0          0          0
         0          0          0          0          0          0          0    
     0  xen-pirq-pcifront-msi  HW_TACHYON
>>
>
The above was from xen-testing, I just switched Ray over to
xen-unstable. He's re-running now with the later hypervisor.

-Bruce
> And you still get on the Xen hypervisor side the DMAR failure of reading
the memory?
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Bruce Edge

2010-Oct-08 17:56 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Fri, Oct 1, 2010 at 2:11 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:> On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge wrote:
>> One of our developers who is working on a tachyon driver is
>> complaining that the pvops domU kernel is not working for these MSI
>> interrupts.
>> This is using the current head of xen/2.6.32.x on both a single
>> Nahelam 920 and a dual E5540. This behavior is consistent with Xen
>> 4.0.1, 4.0.2.rc1-pre and 4.1.
>
>
> I just checked on my SuperMicro X8DTN, this combination
>  - For Dom0, git commit fe999249 (2.6.32.18)
>  - For DomU, devel/xen-pcifront-0.6 or devel/xen-pcifront-0.7
>  - For Hypervisor I used cs 21976, but found that the latest (22155) works
too
>
Booting the above combination of xen/dom0/dpmU logs the following
messages on the dom0 console as soon as domU starts (no custom drivers
loaded yet):

(XEN) tmem: all pools frozen for all domains
(XEN) tmem: all pools thawed for all domains
(XEN) tmem: all pools frozen for all domains
(XEN) tmem: all pools thawed for all domains
mapping kernel into physical memory
about to get started...
(XEN) traps.c:2310:d1 Domain attempted WRMSR 000000000000008b from
0x0000000a00000000 to 0x0000000000000000.
(XEN) traps.c:2310:d1 Domain attempted WRMSR 000000000000008b from
0x0000000a00000000 to 0x0000000000000000.
(XEN) traps.c:2310:d1 Domain attempted WRMSR 000000000000008b from
0x0000000a00000000 to 0x0000000000000000.
(XEN) traps.c:2310:d1 Domain attempted WRMSR 000000000000008b from
0x0000000a00000000 to 0x0000000000000000.
(XEN) traps.c:2310:d1 Domain attempted WRMSR 000000000000008b from
0x0000000a00000000 to 0x0000000000000000.
(XEN) traps.c:2310:d1 Domain attempted WRMSR 000000000000008b from
0x0000000a00000000 to 0x0000000000000000.
[ 1784.608283] ------------[ cut here ]------------
[ 1784.608336] WARNING: at kernel/lockdep.c:2323
trace_hardirqs_on_caller+0x131/0x190()
[ 1784.608418] Hardware name: X8ST3
[ 1784.608445] Modules linked in: xt_physdev ipmi_msghandler ipv6
xenfs xen_gntdev xen_evtchn xen_pciback tun serio_raw joydev bridge
stp llc ioatdma dca usb_storage usbhid hid e1000e
[ 1784.608669] Pid: 11, comm: xenwatch Not tainted
2.6.32.18-pv-ops-stable-debug #1
[ 1784.608725] Call Trace:
[ 1784.608744]  <IRQ>  [<ffffffff81069fbb>]
warn_slowpath_common+0x7b/0xc0
[ 1784.608807]  [<ffffffff815de490>] ? _spin_unlock_irq+0x30/0x40
[ 1784.608853]  [<ffffffff8106a014>] warn_slowpath_null+0x14/0x20
[ 1784.608899]  [<ffffffff810a74b1>] trace_hardirqs_on_caller+0x131/0x190
[ 1784.608945]  [<ffffffff810a751d>] trace_hardirqs_on+0xd/0x10
[ 1784.608991]  [<ffffffff815de490>] _spin_unlock_irq+0x30/0x40
[ 1784.609039]  [<ffffffff813b9736>]
add_to_net_schedule_list_tail+0x86/0xd0
[ 1784.609085]  [<ffffffff813ba948>] netif_be_int+0x38/0x160
[ 1784.609123]  [<ffffffff810dd750>] handle_IRQ_event+0x50/0x160
[ 1784.609170]  [<ffffffff810e05d9>] handle_level_irq+0x99/0x140
[ 1784.609217]  [<ffffffff813adc09>] __xen_evtchn_do_upcall+0x1b9/0x1f0
[ 1784.609263]  [<ffffffff813ae06d>] xen_evtchn_do_upcall+0x3d/0x60
[ 1784.609311]  [<ffffffff8101537e>] xen_do_hypervisor_callback+0x1e/0x30
[ 1784.609356]  <EOI>  [<ffffffff8100940a>] ?
hypercall_page+0x40a/0x1010
[ 1784.609416]  [<ffffffff8100940a>] ? hypercall_page+0x40a/0x1010
[ 1784.609461]  [<ffffffff813b1a13>] ? xb_write+0x103/0x240
[ 1784.609499]  [<ffffffff813b21c0>] ? xs_talkv+0x80/0x1f0
[ 1784.609537]  [<ffffffff813b249b>] ? xs_single+0x4b/0x60
[ 1784.609575]  [<ffffffff813b2b28>] ? xenbus_read+0x48/0x70
[ 1784.609613]  [<ffffffff813bcf6e>] ? frontend_changed+0x47e/0x760
[ 1784.609659]  [<ffffffff813b3e32>] ? xenbus_otherend_changed+0xd2/0x190
[ 1784.609736]  [<ffffffff81010aff>] ? xen_restore_fl_direct_end+0x0/0x1
[ 1784.609782]  [<ffffffff810a991d>] ? lock_release+0xed/0x230
[ 1784.609820]  [<ffffffff813b4540>] ? frontend_changed+0x10/0x20
[ 1784.609866]  [<ffffffff813b1df6>] ? xenwatch_thread+0x56/0x160
[ 1784.609912]  [<ffffffff81090e70>] ? autoremove_wake_function+0x0/0x40
[ 1784.609958]  [<ffffffff813b1da0>] ? xenwatch_thread+0x0/0x160
[ 1784.610004]  [<ffffffff81090b36>] ? kthread+0x96/0xa0
[ 1784.610041]  [<ffffffff8101522a>] ? child_rip+0xa/0x20
[ 1784.610077]  [<ffffffff81014b90>] ? restore_args+0x0/0x30
[ 1784.610114]  [<ffffffff81015220>] ? child_rip+0x0/0x20
[ 1784.610149] ---[ end trace db9e4f4f3b76b033 ]---

-Bruce
> with which where I passed in PCI devices with legacy IRQ, MSI, and MSI-X. I
tried
> a combination of doing this with IOMMU (VT-d) and without - both cases
these devices:
>
> 00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #1 (rev 02)
> 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #2 (rev 02)
> 00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #3 (rev 02)
> 00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI
Controller #1 (rev 02)
> 03:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet
Controller (Copper) (rev 06)
> 0a:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network
Connection (rev 02)
>
> worked just fine (either defining pci=["..."] or just using
pci-attach).
>
> But if I use the latest xen/next or xen/stable-2.6.32.x it does not look
> that happy :-(
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Lin, Ray

2010-Oct-08 21:01 UTC

head link

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

It doesn''t make much difference with xen-unstable. Here is I got from
dom0 after domU was brought up.

(XEN) traps.c:2310:d8 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d8 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d8 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d8 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d8 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d8 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d8 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d8 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d8 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d8 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d8 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d8 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d8 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
(XEN) traps.c:2310:d8 Domain attempted WRMSR 000000000000008b from
0x0000001500000000 to 0x0000000000000000.
[11194.801585] vif8.0: no IPv6 routers present
[11200.947972] blkback: ring-ref 8, event-channel 87, protocol 1 (x86_64-abi)
[11201.106519] pciback 0000:07:00.0: enabling device (0000 -> 0003)
[11201.106578] xen: registering gsi 32 triggering 0 polarity 1
[11201.106598] xen_allocate_pirq: returning irq 32 for gsi 32
[11201.106617] xen: --> irq=32
[11201.106629] Already setup the GSI :32
[11201.106645] pciback 0000:07:00.0: PCI INT A -> GSI 32 (level, low) ->
IRQ 32
[11201.106880] pciback 0000:07:00.0: enabling device (0000 -> 0003)
[11201.106909] xen: registering gsi 32 triggering 0 polarity 1
[11201.106928] xen_allocate_pirq: returning irq 32 for gsi 32
[11201.106947] xen: --> irq=32
[11201.106958] Already setup the GSI :32
[11201.106973] pciback 0000:07:00.0: PCI INT A -> GSI 32 (level, low) ->
IRQ 32
[11201.107083] pciback 0000:07:00.0: PCI INT A disabled
[11201.107244] pciback 0000:07:00.0: enabling device (0000 -> 0003)
[11201.107272] xen: registering gsi 32 triggering 0 polarity 1
[11201.107294] xen_allocate_pirq: returning irq 32 for gsi 32
[11201.107312] xen: --> irq=32
[11201.107353] Already setup the GSI :32
[11201.107370] pciback 0000:07:00.0: PCI INT A -> GSI 32 (level, low) ->
IRQ 32
[11201.107481] pciback 0000:07:00.0: PCI INT A disabled
[11201.107637] pciback 0000:07:00.0: enabling device (0000 -> 0003)
[11201.107665] xen: registering gsi 32 triggering 0 polarity 1
[11201.107684] xen_allocate_pirq: returning irq 32 for gsi 32
[11201.107703] xen: --> irq=32
[11201.107714] Already setup the GSI :32
[11201.107729] pciback 0000:07:00.0: PCI INT A -> GSI 32 (level, low) ->
IRQ 32
[11201.107841] pciback 0000:07:00.0: PCI INT A disabled
[11201.108000] pciback 0000:07:00.0: enabling device (0000 -> 0003)
[11201.108029] xen: registering gsi 32 triggering 0 polarity 1
[11201.108048] xen_allocate_pirq: returning irq 32 for gsi 32
[11201.108066] xen: --> irq=32
[11201.108077] Already setup the GSI :32
[11201.108092] pciback 0000:07:00.0: PCI INT A -> GSI 32 (level, low) ->
IRQ 32
[11201.108292] pciback 0000:07:00.0: PCI INT A disabled
[11201.108471] pciback 0000:07:00.0: enabling device (0000 -> 0003)
[11201.108499] xen: registering gsi 32 triggering 0 polarity 1
[11201.108518] xen_allocate_pirq: returning irq 32 for gsi 32
[11201.108537] xen: --> irq=32
[11201.108548] Already setup the GSI :32
[11201.108563] pciback 0000:07:00.0: PCI INT A -> GSI 32 (level, low) ->
IRQ 32
[11201.109990] pciback 0000:07:00.1: enabling device (0000 -> 0003)
[11201.110020] xen: registering gsi 42 triggering 0 polarity 1
[11201.110040] xen_allocate_pirq: returning irq 42 for gsi 42
[11201.110059] xen: --> irq=42
[11201.110070] Already setup the GSI :42
[11201.110085] pciback 0000:07:00.1: PCI INT B -> GSI 42 (level, low) ->
IRQ 42
[11201.110298] pciback 0000:07:00.1: enabling device (0000 -> 0003)
[11201.110326] xen: registering gsi 42 triggering 0 polarity 1
[11201.110345] xen_allocate_pirq: returning irq 42 for gsi 42
[11201.110364] xen: --> irq=42
[11201.110375] Already setup the GSI :42
[11201.110391] pciback 0000:07:00.1: PCI INT B -> GSI 42 (level, low) ->
IRQ 42
[11201.110492] pciback 0000:07:00.1: PCI INT B disabled
[11201.110636] pciback 0000:07:00.1: enabling device (0000 -> 0003)
[11201.110664] xen: registering gsi 42 triggering 0 polarity 1
[11201.110683] xen_allocate_pirq: returning irq 42 for gsi 42
[11201.110702] xen: --> irq=42
[11201.110713] Already setup the GSI :42
[11201.110728] pciback 0000:07:00.1: PCI INT B -> GSI 42 (level, low) ->
IRQ 42
[11201.110833] pciback 0000:07:00.1: PCI INT B disabled
[11201.111079] pciback 0000:07:00.1: enabling device (0000 -> 0003)
[11201.111109] xen: registering gsi 42 triggering 0 polarity 1
[11201.111128] xen_allocate_pirq: returning irq 42 for gsi 42
[11201.111146] xen: --> irq=42
[11201.111158] Already setup the GSI :42
[11201.111173] pciback 0000:07:00.1: PCI INT B -> GSI 42 (level, low) ->
IRQ 42
[11201.111332] pciback 0000:07:00.1: PCI INT B disabled
[11201.113788] pciback 0000:07:00.1: enabling device (0000 -> 0003)
[11201.113822] xen: registering gsi 42 triggering 0 polarity 1
[11201.113842] xen_allocate_pirq: returning irq 42 for gsi 42
[11201.113860] xen: --> irq=42
[11201.113885] Already setup the GSI :42
[11201.113901] pciback 0000:07:00.1: PCI INT B -> GSI 42 (level, low) ->
IRQ 42
[11201.115796] pciback 0000:07:00.1: PCI INT B disabled
[11201.116421] pciback 0000:07:00.1: enabling device (0000 -> 0003)
[11201.116451] xen: registering gsi 42 triggering 0 polarity 1
[11201.116470] xen_allocate_pirq: returning irq 42 for gsi 42
[11201.116489] xen: --> irq=42
[11201.116532] Already setup the GSI :42
[11201.116548] pciback 0000:07:00.1: PCI INT B -> GSI 42 (level, low) ->
IRQ 42
[11201.121218] pciback 0000:07:00.2: enabling device (0000 -> 0003)
[11201.121251] xen: registering gsi 47 triggering 0 polarity 1
[11201.121270] xen_allocate_pirq: returning irq 47 for gsi 47
[11201.121290] xen: --> irq=47
[11201.121314] Already setup the GSI :47
[11201.121330] pciback 0000:07:00.2: PCI INT C -> GSI 47 (level, low) ->
IRQ 47
[11201.122079] pciback 0000:07:00.2: enabling device (0000 -> 0003)
[11201.122111] xen: registering gsi 47 triggering 0 polarity 1
[11201.122130] xen_allocate_pirq: returning irq 47 for gsi 47
[11201.122149] xen: --> irq=47
[11201.122167] Already setup the GSI :47
[11201.122182] pciback 0000:07:00.2: PCI INT C -> GSI 47 (level, low) ->
IRQ 47
[11201.122305] pciback 0000:07:00.2: PCI INT C disabled
[11201.123677] pciback 0000:07:00.2: enabling device (0000 -> 0003)
[11201.123709] xen: registering gsi 47 triggering 0 polarity 1
[11201.123729] xen_allocate_pirq: returning irq 47 for gsi 47
[11201.123749] xen: --> irq=47
[11201.123774] Already setup the GSI :47
[11201.123790] pciback 0000:07:00.2: PCI INT C -> GSI 47 (level, low) ->
IRQ 47
[11201.123921] pciback 0000:07:00.2: PCI INT C disabled
[11201.124420] pciback 0000:07:00.2: enabling device (0000 -> 0003)
[11201.124451] xen: registering gsi 47 triggering 0 polarity 1
[11201.124470] xen_allocate_pirq: returning irq 47 for gsi 47
[11201.124489] xen: --> irq=47
[11201.124501] Already setup the GSI :47
[11201.124515] pciback 0000:07:00.2: PCI INT C -> GSI 47 (level, low) ->
IRQ 47
[11201.124737] pciback 0000:07:00.2: PCI INT C disabled
[11201.125484] pciback 0000:07:00.2: enabling device (0000 -> 0003)
[11201.125515] xen: registering gsi 47 triggering 0 polarity 1
[11201.125535] xen_allocate_pirq: returning irq 47 for gsi 47
[11201.125555] xen: --> irq=47
[11201.125579] Already setup the GSI :47
[11201.125596] pciback 0000:07:00.2: PCI INT C -> GSI 47 (level, low) ->
IRQ 47
[11201.126228] pciback 0000:07:00.2: PCI INT C disabled
[11201.126448] pciback 0000:07:00.2: enabling device (0000 -> 0003)
[11201.126478] xen: registering gsi 47 triggering 0 polarity 1
[11201.126497] xen_allocate_pirq: returning irq 47 for gsi 47
[11201.126515] xen: --> irq=47
[11201.126527] Already setup the GSI :47
[11201.126542] pciback 0000:07:00.2: PCI INT C -> GSI 47 (level, low) ->
IRQ 47
[11201.129460] pciback 0000:07:00.3: enabling device (0000 -> 0003)
[11201.129492] xen: registering gsi 41 triggering 0 polarity 1
[11201.129511] xen_allocate_pirq: returning irq 41 for gsi 41
[11201.129530] xen: --> irq=41
[11201.129541] Already setup the GSI :41
[11201.129556] pciback 0000:07:00.3: PCI INT D -> GSI 41 (level, low) ->
IRQ 41
[11201.129856] pciback 0000:07:00.3: enabling device (0000 -> 0003)
[11201.129886] xen: registering gsi 41 triggering 0 polarity 1
[11201.129906] xen_allocate_pirq: returning irq 41 for gsi 41
[11201.129924] xen: --> irq=41
[11201.129936] Already setup the GSI :41
[11201.129951] pciback 0000:07:00.3: PCI INT D -> GSI 41 (level, low) ->
IRQ 41
[11201.130277] pciback 0000:07:00.3: PCI INT D disabled
[11201.130449] pciback 0000:07:00.3: enabling device (0000 -> 0003)
[11201.130478] xen: registering gsi 41 triggering 0 polarity 1
[11201.130497] xen_allocate_pirq: returning irq 41 for gsi 41
[11201.130516] xen: --> irq=41
[11201.130528] Already setup the GSI :41
[11201.130543] pciback 0000:07:00.3: PCI INT D -> GSI 41 (level, low) ->
IRQ 41
[11201.130871] pciback 0000:07:00.3: PCI INT D disabled
[11201.131025] pciback 0000:07:00.3: enabling device (0000 -> 0003)
[11201.131054] xen: registering gsi 41 triggering 0 polarity 1
[11201.131073] xen_allocate_pirq: returning irq 41 for gsi 41
[11201.131092] xen: --> irq=41
[11201.131103] Already setup the GSI :41
[11201.131119] pciback 0000:07:00.3: PCI INT D -> GSI 41 (level, low) ->
IRQ 41
[11201.131465] pciback 0000:07:00.3: PCI INT D disabled
[11201.132493] pciback 0000:07:00.3: enabling device (0000 -> 0003)
[11201.132524] xen: registering gsi 41 triggering 0 polarity 1
[11201.132543] xen_allocate_pirq: returning irq 41 for gsi 41
[11201.132562] xen: --> irq=41
[11201.132574] Already setup the GSI :41
[11201.132591] pciback 0000:07:00.3: PCI INT D -> GSI 41 (level, low) ->
IRQ 41
[11201.132992] pciback 0000:07:00.3: PCI INT D disabled
[11201.133155] pciback 0000:07:00.3: enabling device (0000 -> 0003)
[11201.133185] xen: registering gsi 41 triggering 0 polarity 1
[11201.133204] xen_allocate_pirq: returning irq 41 for gsi 41
[11201.133223] xen: --> irq=41
[11201.133234] Already setup the GSI :41
[11201.133249] pciback 0000:07:00.3: PCI INT D -> GSI 41 (level, low) ->
IRQ 41
[11248.058409] pciback 0000:07:00.0: setting latency timer to 64
[11248.059095]   alloc irq_desc for 458 on node 0
[11248.059120]   alloc kstat_irqs on node 0
[11248.164759] pciback 0000:07:00.1: setting latency timer to 64
[11248.165337]   alloc irq_desc for 457 on node 0
[11248.165362]   alloc kstat_irqs on node 0
[11248.263380] pciback 0000:07:00.2: setting latency timer to 64
[11248.263993]   alloc irq_desc for 456 on node 0
[11248.264018]   alloc kstat_irqs on node 0
[11248.527645] pciback 0000:07:00.3: setting latency timer to 64
[11248.528260]   alloc irq_desc for 455 on node 0
[11248.528287]   alloc kstat_irqs on node 0

 

-----Original Message-----
From: Bruce Edge [mailto:bruce.edge@gmail.com] 
Sent: Friday, October 08, 2010 10:52 AM
To: Konrad Rzeszutek Wilk
Cc: Lin, Ray; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Fri, Oct 8, 2010 at 10:30 AM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:> On Fri, Oct 08, 2010 at 10:48:01AM -0600, Lin, Ray wrote:
>>
>> I just tried Bruce''s latest kernel build based on
Konrad''s devel/xen-pcifront-0.7. It doesn''t help the issue we
have. The driver still doesn''t recognize the source of interrupt, even
though the interrupts happen.
>>
>>
>> 124:      87792          0          0          0          0          
>> 0      12208          0          0          0          0          0    
     
>> 0          0  xen-pirq-pcifront-msi  HW_TACHYON
>> 125:      89692          0          0          0      10308          
>> 0          0          0          0          0          0          0    
     
>> 0          0  xen-pirq-pcifront-msi  HW_TACHYON
>> 126:      90979          0       9021          0          0          
>> 0          0          0          0          0          0          0    
     
>> 0          0  xen-pirq-pcifront-msi  HW_TACHYON
>> 127:     100000          0          0          0          0          
>> 0          0          0          0          0          0          0    
     
>> 0          0  xen-pirq-pcifront-msi  HW_TACHYON
>>
>
The above was from xen-testing, I just switched Ray over to xen-unstable.
He''s re-running now with the later hypervisor.

-Bruce
> And you still get on the Xen hypervisor side the DMAR failure of reading
the memory?
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Bruce Edge

2010-Oct-11 21:12 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Fri, Oct 1, 2010 at 2:11 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:> On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge wrote:
>> One of our developers who is working on a tachyon driver is
>> complaining that the pvops domU kernel is not working for these MSI
>> interrupts.
>> This is using the current head of xen/2.6.32.x on both a single
>> Nahelam 920 and a dual E5540. This behavior is consistent with Xen
>> 4.0.1, 4.0.2.rc1-pre and 4.1.
>
>
> I just checked on my SuperMicro X8DTN, this combination
>  - For Dom0, git commit fe999249 (2.6.32.18)
>  - For DomU, devel/xen-pcifront-0.6 or devel/xen-pcifront-0.7
>  - For Hypervisor I used cs 21976, but found that the latest (22155) works
too
>
> with which where I passed in PCI devices with legacy IRQ, MSI, and MSI-X. I
tried
> a combination of doing this with IOMMU (VT-d) and without - both cases
these devices:
>
> 00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #1 (rev 02)
> 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #2 (rev 02)
> 00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #3 (rev 02)
> 00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI
Controller #1 (rev 02)
> 03:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet
Controller (Copper) (rev 06)
> 0a:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network
Connection (rev 02)
>
> worked just fine (either defining pci=["..."] or just using
pci-attach).
>
> But if I use the latest xen/next or xen/stable-2.6.32.x it does not look
> that happy :-(
>
>
Konrad,
To try eliminate the remaining differences here, could you post your
dom0/domU config files?
I''d like to build the same kernels to get an apples->apples
comparison.

Also, could you include your grub info and domU cfg file?

These may eliminate some of the remaining diffs in the configs and
show why your''s works while mine does not.

Thanks

-Bruce

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk

2010-Oct-11 21:46 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Mon, Oct 11, 2010 at 02:12:22PM -0700, Bruce Edge
wrote:> On Fri, Oct 1, 2010 at 2:11 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge wrote:
> >> One of our developers who is working on a tachyon driver is
> >> complaining that the pvops domU kernel is not working for these
MSI
> >> interrupts.
> >> This is using the current head of xen/2.6.32.x on both a single
> >> Nahelam 920 and a dual E5540. This behavior is consistent with Xen
> >> 4.0.1, 4.0.2.rc1-pre and 4.1.
> >
> >
> > I just checked on my SuperMicro X8DTN, this combination
> >  - For Dom0, git commit fe999249 (2.6.32.18)
> >  - For DomU, devel/xen-pcifront-0.6 or devel/xen-pcifront-0.7
> >  - For Hypervisor I used cs 21976, but found that the latest (22155)
works too
> >
> > with which where I passed in PCI devices with legacy IRQ, MSI, and
MSI-X. I tried
> > a combination of doing this with IOMMU (VT-d) and without - both cases
these devices:
> >
> > 00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB
UHCI Controller #1 (rev 02)
> > 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB
UHCI Controller #2 (rev 02)
> > 00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB
UHCI Controller #3 (rev 02)
> > 00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2
EHCI Controller #1 (rev 02)
> > 03:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit
Ethernet Controller (Copper) (rev 06)
> > 0a:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network
Connection (rev 02)
> >
> > worked just fine (either defining pci=["..."] or just using
pci-attach).
> >
> > But if I use the latest xen/next or xen/stable-2.6.32.x it does not
look
> > that happy :-(
> >
> >
> 
> Konrad,
> To try eliminate the remaining differences here, could you post your
> dom0/domU config files?
Sure. See attached> I''d like to build the same kernels to get an apples->apples
comparison.
> 
> Also, could you include your grub info and domU cfg file?
> 
> These may eliminate some of the remaining diffs in the configs and
> show why your''s works while mine does not.
> 
> Thanks
> 
> -Bruce






_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Bruce Edge

2010-Oct-13 21:36 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Mon, Oct 11, 2010 at 2:46 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:> On Mon, Oct 11, 2010 at 02:12:22PM -0700, Bruce Edge wrote:
>> On Fri, Oct 1, 2010 at 2:11 PM, Konrad Rzeszutek Wilk
>> <konrad.wilk@oracle.com> wrote:
>> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge wrote:
>> >> One of our developers who is working on a tachyon driver is
>> >> complaining that the pvops domU kernel is not working for
these MSI
>> >> interrupts.
>> >> This is using the current head of xen/2.6.32.x on both a
single
>> >> Nahelam 920 and a dual E5540. This behavior is consistent with
Xen
>> >> 4.0.1, 4.0.2.rc1-pre and 4.1.
>> >
>> >
>> > I just checked on my SuperMicro X8DTN, this combination
>> >  - For Dom0, git commit fe999249 (2.6.32.18)
>> >  - For DomU, devel/xen-pcifront-0.6 or devel/xen-pcifront-0.7
>> >  - For Hypervisor I used cs 21976, but found that the latest
(22155) works too
>> >
>> > with which where I passed in PCI devices with legacy IRQ, MSI, and
MSI-X. I tried
>> > a combination of doing this with IOMMU (VT-d) and without - both
cases these devices:
>> >
>> > 00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB
UHCI Controller #1 (rev 02)
>> > 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB
UHCI Controller #2 (rev 02)
>> > 00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB
UHCI Controller #3 (rev 02)
>> > 00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family)
USB2 EHCI Controller #1 (rev 02)
>> > 03:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit
Ethernet Controller (Copper) (rev 06)
>> > 0a:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit
Network Connection (rev 02)
>> >
>> > worked just fine (either defining pci=["..."] or just
using pci-attach).
>> >
>> > But if I use the latest xen/next or xen/stable-2.6.32.x it does
not look
>> > that happy :-(
>> >
>> >
>>
>> Konrad,
>> To try eliminate the remaining differences here, could you post your
>> dom0/domU config files?
>
> Sure. See attached
Konrad,
That made a big difference. Looks much better now. It''s been kicked
over to several developers who have each got our tachyon driver
working a little bit better.

Now the sticking point is an apparent limitation on the amount of
memory one can request using pci_map_single. It appears that we can
only ask for 256K or less.  We need a 2MB DMA buffer.
Is there some alternate mechanism for getting a larger physically
contiguous buffer under pvops?

Thanks

-Bruce
>> I''d like to build the same kernels to get an apples->apples
comparison.
>>
>> Also, could you include your grub info and domU cfg file?
>>
>> These may eliminate some of the remaining diffs in the configs and
>> show why your''s works while mine does not.
>>
>> Thanks
>>
>> -Bruce
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk

2010-Oct-13 21:46 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Wed, Oct 13, 2010 at 02:36:47PM -0700, Bruce Edge
wrote:> On Mon, Oct 11, 2010 at 2:46 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > On Mon, Oct 11, 2010 at 02:12:22PM -0700, Bruce Edge wrote:
> >> On Fri, Oct 1, 2010 at 2:11 PM, Konrad Rzeszutek Wilk
> >> <konrad.wilk@oracle.com> wrote:
> >> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge wrote:
> >> >> One of our developers who is working on a tachyon driver
is
> >> >> complaining that the pvops domU kernel is not working for
these MSI
> >> >> interrupts.
> >> >> This is using the current head of xen/2.6.32.x on both a
single
> >> >> Nahelam 920 and a dual E5540. This behavior is consistent
with Xen
> >> >> 4.0.1, 4.0.2.rc1-pre and 4.1.
> >> >
> >> >
> >> > I just checked on my SuperMicro X8DTN, this combination
> >> >  - For Dom0, git commit fe999249 (2.6.32.18)
> >> >  - For DomU, devel/xen-pcifront-0.6 or devel/xen-pcifront-0.7
> >> >  - For Hypervisor I used cs 21976, but found that the latest
(22155) works too
> >> >
> >> > with which where I passed in PCI devices with legacy IRQ,
MSI, and MSI-X. I tried
> >> > a combination of doing this with IOMMU (VT-d) and without -
both cases these devices:
> >> >
> >> > 00:1d.0 USB Controller: Intel Corporation 82801I (ICH9
Family) USB UHCI Controller #1 (rev 02)
> >> > 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9
Family) USB UHCI Controller #2 (rev 02)
> >> > 00:1d.2 USB Controller: Intel Corporation 82801I (ICH9
Family) USB UHCI Controller #3 (rev 02)
> >> > 00:1d.7 USB Controller: Intel Corporation 82801I (ICH9
Family) USB2 EHCI Controller #1 (rev 02)
> >> > 03:00.0 Ethernet controller: Intel Corporation 82572EI
Gigabit Ethernet Controller (Copper) (rev 06)
> >> > 0a:00.1 Ethernet controller: Intel Corporation 82575EB
Gigabit Network Connection (rev 02)
> >> >
> >> > worked just fine (either defining pci=["..."] or
just using pci-attach).
> >> >
> >> > But if I use the latest xen/next or xen/stable-2.6.32.x it
does not look
> >> > that happy :-(
> >> >
> >> >
> >>
> >> Konrad,
> >> To try eliminate the remaining differences here, could you post
your
> >> dom0/domU config files?
> >
> > Sure. See attached
> 
> Konrad,
> That made a big difference. Looks much better now. It''s been
kicked
> over to several developers who have each got our tachyon driver
> working a little bit better.
Hmm, I didn''t really try to do anything fancy with the configs. Any
inklings of what config option might have caused all this headache?
> 
> Now the sticking point is an apparent limitation on the amount of
> memory one can request using pci_map_single. It appears that we can
> only ask for 256K or less.  We need a 2MB DMA buffer.
You can''t use SG buffers? And chain them together and provide them
to the device? A lot of other drivers do this ...
> Is there some alternate mechanism for getting a larger physically
> contiguous buffer under pvops?
Why the contiguous requirement?> 
> Thanks
> 
> -Bruce
> 
> >> I''d like to build the same kernels to get an
apples->apples comparison.
> >>
> >> Also, could you include your grub info and domU cfg file?
> >>
> >> These may eliminate some of the remaining diffs in the configs and
> >> show why your''s works while mine does not.
> >>
> >> Thanks
> >>
> >> -Bruce
> >
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Bruce Edge

2010-Oct-13 22:00 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Wed, Oct 13, 2010 at 2:46 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:> On Wed, Oct 13, 2010 at 02:36:47PM -0700, Bruce Edge wrote:
>> On Mon, Oct 11, 2010 at 2:46 PM, Konrad Rzeszutek Wilk
>> <konrad.wilk@oracle.com> wrote:
>> > On Mon, Oct 11, 2010 at 02:12:22PM -0700, Bruce Edge wrote:
>> >> On Fri, Oct 1, 2010 at 2:11 PM, Konrad Rzeszutek Wilk
>> >> <konrad.wilk@oracle.com> wrote:
>> >> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge
wrote:
>> >> >> One of our developers who is working on a tachyon
driver is
>> >> >> complaining that the pvops domU kernel is not working
for these MSI
>> >> >> interrupts.
>> >> >> This is using the current head of xen/2.6.32.x on
both a single
>> >> >> Nahelam 920 and a dual E5540. This behavior is
consistent with Xen
>> >> >> 4.0.1, 4.0.2.rc1-pre and 4.1.
>> >> >
>> >> >
>> >> > I just checked on my SuperMicro X8DTN, this combination
>> >> >  - For Dom0, git commit fe999249 (2.6.32.18)
>> >> >  - For DomU, devel/xen-pcifront-0.6 or
devel/xen-pcifront-0.7
>> >> >  - For Hypervisor I used cs 21976, but found that the
latest (22155) works too
>> >> >
>> >> > with which where I passed in PCI devices with legacy IRQ,
MSI, and MSI-X. I tried
>> >> > a combination of doing this with IOMMU (VT-d) and without
- both cases these devices:
>> >> >
>> >> > 00:1d.0 USB Controller: Intel Corporation 82801I (ICH9
Family) USB UHCI Controller #1 (rev 02)
>> >> > 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9
Family) USB UHCI Controller #2 (rev 02)
>> >> > 00:1d.2 USB Controller: Intel Corporation 82801I (ICH9
Family) USB UHCI Controller #3 (rev 02)
>> >> > 00:1d.7 USB Controller: Intel Corporation 82801I (ICH9
Family) USB2 EHCI Controller #1 (rev 02)
>> >> > 03:00.0 Ethernet controller: Intel Corporation 82572EI
Gigabit Ethernet Controller (Copper) (rev 06)
>> >> > 0a:00.1 Ethernet controller: Intel Corporation 82575EB
Gigabit Network Connection (rev 02)
>> >> >
>> >> > worked just fine (either defining pci=["..."]
or just using pci-attach).
>> >> >
>> >> > But if I use the latest xen/next or xen/stable-2.6.32.x
it does not look
>> >> > that happy :-(
>> >> >
>> >> >
>> >>
>> >> Konrad,
>> >> To try eliminate the remaining differences here, could you
post your
>> >> dom0/domU config files?
>> >
>> > Sure. See attached
>>
>> Konrad,
>> That made a big difference. Looks much better now. It''s been
kicked
>> over to several developers who have each got our tachyon driver
>> working a little bit better.
>
> Hmm, I didn''t really try to do anything fancy with the configs.
Any
> inklings of what config option might have caused all this headache?
I had pruned out a lot of stuff I didn''t need from the kernel. I may
have been a bit overzealous. Also, we were using the same kernel for
dom0 and domU. It may be that having all the dom0 baggage in the domU
has some unexpected consequences. Very vague I know, sorry. If I have
time I''ll try narrow down the diffs that break it.
>
>>
>> Now the sticking point is an apparent limitation on the amount of
>> memory one can request using pci_map_single. It appears that we can
>> only ask for 256K or less.  We need a 2MB DMA buffer.
>
> You can''t use SG buffers? And chain them together and provide them
> to the device? A lot of other drivers do this ...
Perhaps. I don''t know. I''m not the driver author. I think that
it
worked using 1 giant buffer before so there was no need to use SG
buffers.
>
>> Is there some alternate mechanism for getting a larger physically
>> contiguous buffer under pvops?
>
> Why the contiguous requirement?
It may not be a  hard requirement and was just a matter of
convenience. I''ll kick that over the the tachyon guys and see what
they say. I think it''s a non-trivial exercise to switch over. All the
internal buffer mgmt would have to change. If that''s the only option,
well, that''s it. ...is it?

Thanks again for your help. You''ve been instrumental it getting this
up.

-Bruce
>>
>> Thanks
>>
>> -Bruce
>>
>> >> I''d like to build the same kernels to get an
apples->apples comparison.
>> >>
>> >> Also, could you include your grub info and domU cfg file?
>> >>
>> >> These may eliminate some of the remaining diffs in the configs
and
>> >> show why your''s works while mine does not.
>> >>
>> >> Thanks
>> >>
>> >> -Bruce
>> >
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Bruce Edge

2010-Oct-13 22:08 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

On Wed, Oct 13, 2010 at 3:00 PM, Bruce Edge <bruce.edge@gmail.com>
wrote:> On Wed, Oct 13, 2010 at 2:46 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
>> On Wed, Oct 13, 2010 at 02:36:47PM -0700, Bruce Edge wrote:
>>> On Mon, Oct 11, 2010 at 2:46 PM, Konrad Rzeszutek Wilk
>>> <konrad.wilk@oracle.com> wrote:
>>> > On Mon, Oct 11, 2010 at 02:12:22PM -0700, Bruce Edge wrote:
>>> >> On Fri, Oct 1, 2010 at 2:11 PM, Konrad Rzeszutek Wilk
>>> >> <konrad.wilk@oracle.com> wrote:
>>> >> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge
wrote:
>>> >> >> One of our developers who is working on a tachyon
driver is
>>> >> >> complaining that the pvops domU kernel is not
working for these MSI
>>> >> >> interrupts.
>>> >> >> This is using the current head of xen/2.6.32.x on
both a single
>>> >> >> Nahelam 920 and a dual E5540. This behavior is
consistent with Xen
>>> >> >> 4.0.1, 4.0.2.rc1-pre and 4.1.
>>> >> >
>>> >> >
>>> >> > I just checked on my SuperMicro X8DTN, this
combination
>>> >> >  - For Dom0, git commit fe999249 (2.6.32.18)
>>> >> >  - For DomU, devel/xen-pcifront-0.6 or
devel/xen-pcifront-0.7
>>> >> >  - For Hypervisor I used cs 21976, but found that the
latest (22155) works too
>>> >> >
>>> >> > with which where I passed in PCI devices with legacy
IRQ, MSI, and MSI-X. I tried
>>> >> > a combination of doing this with IOMMU (VT-d) and
without - both cases these devices:
>>> >> >
>>> >> > 00:1d.0 USB Controller: Intel Corporation 82801I
(ICH9 Family) USB UHCI Controller #1 (rev 02)
>>> >> > 00:1d.1 USB Controller: Intel Corporation 82801I
(ICH9 Family) USB UHCI Controller #2 (rev 02)
>>> >> > 00:1d.2 USB Controller: Intel Corporation 82801I
(ICH9 Family) USB UHCI Controller #3 (rev 02)
>>> >> > 00:1d.7 USB Controller: Intel Corporation 82801I
(ICH9 Family) USB2 EHCI Controller #1 (rev 02)
>>> >> > 03:00.0 Ethernet controller: Intel Corporation
82572EI Gigabit Ethernet Controller (Copper) (rev 06)
>>> >> > 0a:00.1 Ethernet controller: Intel Corporation
82575EB Gigabit Network Connection (rev 02)
>>> >> >
>>> >> > worked just fine (either defining
pci=["..."] or just using pci-attach).
>>> >> >
>>> >> > But if I use the latest xen/next or
xen/stable-2.6.32.x it does not look
>>> >> > that happy :-(
>>> >> >
>>> >> >
>>> >>
>>> >> Konrad,
>>> >> To try eliminate the remaining differences here, could you
post your
>>> >> dom0/domU config files?
>>> >
>>> > Sure. See attached
>>>
>>> Konrad,
>>> That made a big difference. Looks much better now. It''s
been kicked
>>> over to several developers who have each got our tachyon driver
>>> working a little bit better.
>>
>> Hmm, I didn''t really try to do anything fancy with the
configs. Any
>> inklings of what config option might have caused all this headache?
>
> I had pruned out a lot of stuff I didn''t need from the kernel. I
may
> have been a bit overzealous. Also, we were using the same kernel for
> dom0 and domU. It may be that having all the dom0 baggage in the domU
> has some unexpected consequences. Very vague I know, sorry. If I have
> time I''ll try narrow down the diffs that break it.
>
>>
>>>
>>> Now the sticking point is an apparent limitation on the amount of
>>> memory one can request using pci_map_single. It appears that we can
>>> only ask for 256K or less.  We need a 2MB DMA buffer.
>>
>> You can''t use SG buffers? And chain them together and provide
them
>> to the device? A lot of other drivers do this ...
>
> Perhaps. I don''t know. I''m not the driver author. I think
that it
> worked using 1 giant buffer before so there was no need to use SG
> buffers.
Got some more info. It''s a hardware requirement for the tachyon chip.
It uses a 2 MB block for the FC SEST index table. It''s one place where
we can''t use SG.
It really not a matter of laziness.

-Bruce
>
>>
>>> Is there some alternate mechanism for getting a larger physically
>>> contiguous buffer under pvops?
>>
>> Why the contiguous requirement?
>
> It may not be a  hard requirement and was just a matter of
> convenience. I''ll kick that over the the tachyon guys and see what
> they say. I think it''s a non-trivial exercise to switch over. All
the
> internal buffer mgmt would have to change. If that''s the only
option,
> well, that''s it. ...is it?
>
> Thanks again for your help. You''ve been instrumental it getting
this up.
>
> -Bruce
>
>>>
>>> Thanks
>>>
>>> -Bruce
>>>
>>> >> I''d like to build the same kernels to get an
apples->apples comparison.
>>> >>
>>> >> Also, could you include your grub info and domU cfg file?
>>> >>
>>> >> These may eliminate some of the remaining diffs in the
configs and
>>> >> show why your''s works while mine does not.
>>> >>
>>> >> Thanks
>>> >>
>>> >> -Bruce
>>> >
>>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk

2010-Oct-14 13:57 UTC

head link

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

> >> You can''t use SG buffers? And chain them together and
provide them
> >> to the device? A lot of other drivers do this ...
> >
> > Perhaps. I don''t know. I''m not the driver author. I
think that it
> > worked using 1 giant buffer before so there was no need to use SG
> > buffers.
> 
> Got some more info. It''s a hardware requirement for the tachyon
chip.
> It uses a 2 MB block for the FC SEST index table. It''s one place
where
> we can''t use SG.
OK, pci_alloc_consistent is the way. I think you can go up to 2MB with it.
> It really not a matter of laziness.
<nods>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Xen devel - Sep 2010 - pv-ops domU not working with MSI interrupts on Nehalem

[Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

RE: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem

Re: [Xen-devel] pv-ops domU not working with MSI interrupts on Nehalem