Yuji Shimada
2008-Aug-29 08:05 UTC
[Xen-devel] FC-HBA assigned to guest domain does not work.
I assigned FC-HBA to guest domain, but it did not work.
FC-HBA seems to write its internal memory which is mapped to host
memory space via pci transaction. But there is no mapping in IOMMU''s
page table, so that page fault occurs in IOMMU.
I think that MMIO resource mapped via p2m table should be mapped via
IOMMU''s page table too. In other word, XEN_DOMCTL_memory_mapping
hypercall should update not only p2m table, but also IOMMU''s page
table.
What do you think about this?
Does anyone see the same issue?
/var/log/message on guest linux:
Aug 28 11:53:28 localhost kernel: Emulex LightPulse Fibre Channel SCSI driver
8.2.0.22
Aug 28 11:53:28 localhost kernel: Copyright(c) 2004-2008 Emulex. All rights
reserved.
Aug 28 11:53:28 localhost kernel: GSI 16 sharing vector 0xA9 and IRQ 16
Aug 28 11:53:28 localhost kernel: ACPI: PCI Interrupt 0000:00:06.0[A] -> GSI
40(level, low) -> IRQ 169
Aug 28 11:53:28 localhost kernel: scsi0 : on PCI bus 00 device 30 irq 169
Aug 28 11:53:28 localhost kernel: lpfc 0000:00:06.0: 0:0453 Adapter failed to
init, mbxCmd xb READ_CONFIG, mbxStatus x0
Aug 28 11:53:28 localhost kernel: ACPI: PCI interrupt for device 0000:00:06.0
disabled
lspci output on guest linux:
00:06.0 Fibre Channel: Emulex Corporation Zephyr LightPulse Fibre Channel Host
Adapter (rev 02)
Subsystem: Emulex Corporation Zephyr LightPulse Fibre Channel Host
Adapter
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort+ >SERR- <PERR-
Interrupt: pin A routed to IRQ 169
Region 0: Memory at f3041000 (64-bit, non-prefetchable) [disabled]
[size=4K]
^^^^^^^^
Region 2: Memory at f3042000 (64-bit, non-prefetchable) [disabled]
[size=4K]
Region 4: I/O ports at c100 [disabled] [size=256]
Expansion ROM at f3000000 [disabled] [size=256K]
xm dmesg output on dom0 linux:
(XEN) [VT-D]iommu.c:775: iommu_page_fault: iommu->reg = ffff828bfff58000
(XEN) [VT-D]iommu.c:747: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:729: iommu_fault:DMA Write: 7:0.0 addr f3041000 REASON 5
^^^^^^^^
iommu->reg = ffff828bfff58000
(XEN) print_vtd_entries: iommu = ffff8300d6e01780 bdf = 7:0:0 gmfn = f3041
(XEN) root_entry = ffff83021fdf2000
(XEN) root_entry[7] = 217de9001
(XEN) context = ffff830217de9000
(XEN) context[0] = 201_2155d1001
(XEN) l3 = ffff8302155d1000
(XEN) l3_index = 3
(XEN) l3[3] = 1d802a003
(XEN) l2 = ffff8301d802a000
(XEN) l2_index = 198
(XEN) l2[198] = 0
(XEN) l2[198] not present
Thanks,
--
Yuji Shimada
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Cui, Dexuan
2008-Aug-29 09:40 UTC
RE: [Xen-devel] FC-HBA assigned to guest domain does not work.
I''m also suffering from the issue: the assigned FC-HBA (rev 03)
doesn''t work.
But I don''t see the iommu fault in your mail.
Serial messages don''t give any special info in my case.
In my case, the issue looks like a regression. For example, "Xen 16711 +
Dom0 417" can work while the latest xen-unstable doesn''t work.
I can only access the host remotely and it died... I''ll continue to
look into the issue next week.
-- Dexuan
-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Yuji Shimada
Sent: 2008年8月29日 16:05
To: xen-devel@lists.xensource.com
Subject: [Xen-devel] FC-HBA assigned to guest domain does not work.
I assigned FC-HBA to guest domain, but it did not work.
FC-HBA seems to write its internal memory which is mapped to host
memory space via pci transaction. But there is no mapping in IOMMU''s
page table, so that page fault occurs in IOMMU.
I think that MMIO resource mapped via p2m table should be mapped via
IOMMU''s page table too. In other word, XEN_DOMCTL_memory_mapping
hypercall should update not only p2m table, but also IOMMU''s page
table.
What do you think about this?
Does anyone see the same issue?
/var/log/message on guest linux:
Aug 28 11:53:28 localhost kernel: Emulex LightPulse Fibre Channel SCSI driver
8.2.0.22
Aug 28 11:53:28 localhost kernel: Copyright(c) 2004-2008 Emulex. All rights
reserved.
Aug 28 11:53:28 localhost kernel: GSI 16 sharing vector 0xA9 and IRQ 16
Aug 28 11:53:28 localhost kernel: ACPI: PCI Interrupt 0000:00:06.0[A] -> GSI
40(level, low) -> IRQ 169
Aug 28 11:53:28 localhost kernel: scsi0 : on PCI bus 00 device 30 irq 169
Aug 28 11:53:28 localhost kernel: lpfc 0000:00:06.0: 0:0453 Adapter failed to
init, mbxCmd xb READ_CONFIG, mbxStatus x0
Aug 28 11:53:28 localhost kernel: ACPI: PCI interrupt for device 0000:00:06.0
disabled
lspci output on guest linux:
00:06.0 Fibre Channel: Emulex Corporation Zephyr LightPulse Fibre Channel Host
Adapter (rev 02)
Subsystem: Emulex Corporation Zephyr LightPulse Fibre Channel Host
Adapter
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort+ >SERR- <PERR-
Interrupt: pin A routed to IRQ 169
Region 0: Memory at f3041000 (64-bit, non-prefetchable) [disabled]
[size=4K]
^^^^^^^^
Region 2: Memory at f3042000 (64-bit, non-prefetchable) [disabled]
[size=4K]
Region 4: I/O ports at c100 [disabled] [size=256]
Expansion ROM at f3000000 [disabled] [size=256K]
xm dmesg output on dom0 linux:
(XEN) [VT-D]iommu.c:775: iommu_page_fault: iommu->reg = ffff828bfff58000
(XEN) [VT-D]iommu.c:747: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:729: iommu_fault:DMA Write: 7:0.0 addr f3041000 REASON 5
^^^^^^^^
iommu->reg = ffff828bfff58000
(XEN) print_vtd_entries: iommu = ffff8300d6e01780 bdf = 7:0:0 gmfn = f3041
(XEN) root_entry = ffff83021fdf2000
(XEN) root_entry[7] = 217de9001
(XEN) context = ffff830217de9000
(XEN) context[0] = 201_2155d1001
(XEN) l3 = ffff8302155d1000
(XEN) l3_index = 3
(XEN) l3[3] = 1d802a003
(XEN) l2 = ffff8301d802a000
(XEN) l2_index = 198
(XEN) l2[198] = 0
(XEN) l2[198] not present
Thanks,
--
Yuji Shimada
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Yuji Shimada
2008-Sep-02 05:32 UTC
Re: [Xen-devel] FC-HBA assigned to guest domain does not work.
FC-HBA''s MMIO resource are mapped to guest physical address and machine address. I found both addresses need to be the same. I created the series of patches which map MMIO resource to the same address in guest physical address space and in machine address space. After applying this patch, my FC-HBA worked and IOMMU page fault did not occur. The patches are for only discussion. I''d like comments of developers.> I think that MMIO resource mapped via p2m table should be mapped via > IOMMU''s page table too. In other word, XEN_DOMCTL_memory_mapping > hypercall should update not only p2m table, but also IOMMU''s page table.Currently I don''t think this is required. Thanks, Signed-off-by: Yuji Shimada <shimada-yxb@necst.nec.co.jp> -- Yuji Shimada _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Cui, Dexuan
2008-Sep-02 07:04 UTC
RE: [Xen-devel] FC-HBA assigned to guest domain does not work.
I tried the patches and found they can''t fix the issue I meet with. (As I mentioned previously, my symptom is different from yours) Maybe there are 2 different issues. Thanks, -- Dexuan -----Original Message----- From: Yuji Shimada [mailto:shimada-yxb@necst.nec.co.jp] Sent: 2008年9月2日 13:33 To: Cui, Dexuan; xen-devel@lists.xensource.com Subject: Re: [Xen-devel] FC-HBA assigned to guest domain does not work. FC-HBA''s MMIO resource are mapped to guest physical address and machine address. I found both addresses need to be the same. I created the series of patches which map MMIO resource to the same address in guest physical address space and in machine address space. After applying this patch, my FC-HBA worked and IOMMU page fault did not occur. The patches are for only discussion. I''d like comments of developers.> I think that MMIO resource mapped via p2m table should be mapped via > IOMMU''s page table too. In other word, XEN_DOMCTL_memory_mapping > hypercall should update not only p2m table, but also IOMMU''s page table.Currently I don''t think this is required. Thanks, Signed-off-by: Yuji Shimada <shimada-yxb@necst.nec.co.jp> -- Yuji Shimada _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Sep-02 07:38 UTC
Re: [Xen-devel] FC-HBA assigned to guest domain does not work.
The hvmloader side is perhaps plausible, since it only changes its behaviour if qemu exposes an already-configured BAR. Fair enough. I don''t think that blanket always-use-physical-BAR-address is a great idea in qemu -- perhaps if it could be configurable that would be okay. -- Keir On 2/9/08 06:32, "Yuji Shimada" <shimada-yxb@necst.nec.co.jp> wrote:> FC-HBA''s MMIO resource are mapped to guest physical address and > machine address. I found both addresses need to be the same. > > I created the series of patches which map MMIO resource to the same > address in guest physical address space and in machine address space. > After applying this patch, my FC-HBA worked and IOMMU page fault did > not occur. > > The patches are for only discussion. > I''d like comments of developers. > > >> I think that MMIO resource mapped via p2m table should be mapped via >> IOMMU''s page table too. In other word, XEN_DOMCTL_memory_mapping >> hypercall should update not only p2m table, but also IOMMU''s page table. > > Currently I don''t think this is required. > > Thanks, > > Signed-off-by: Yuji Shimada <shimada-yxb@necst.nec.co.jp> > > -- > Yuji Shimada > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yuji Shimada
2008-Sep-03 07:54 UTC
Re: [Xen-devel] FC-HBA assigned to guest domain does not work.
Thank you for your reply. My patches say that some devices which require that emulated BAR address is equal to actual BAR address. I don''t have internal design specification of FC-HBA. But I presume FC-HBA''s device driver and firmware communicate via its internal memory. And they use the same address to access internal memory. On non-virtualized environment, device driver and firmware use the same physical address. So they can communicate each other. On virtualized environment, device driver use guest physical address, while firmware use machine address. So they can''t communicate. My patches are not good solution to fix the issue. Hot-added device will not work, because Guest OS configures BARs. And other changes will be needed. For example, we need to change guest memory map to avoid conflict between MMIO resource and memory. Additionally guest physical address should be independent of machine address. I don''t have a good idea to solve the issue, currently. If VT-d provides some method to solve the issue, it is nice. -- Yuji Shimada On Tue, 02 Sep 2008 08:38:55 +0100 Keir Fraser <keir.fraser@eu.citrix.com> wrote:> The hvmloader side is perhaps plausible, since it only changes its behaviour > if qemu exposes an already-configured BAR. Fair enough. I don''t think that > blanket always-use-physical-BAR-address is a great idea in qemu -- perhaps > if it could be configurable that would be okay. > > -- Keir_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yuji Shimada
2008-Sep-03 08:17 UTC
Re: [Xen-devel] FC-HBA assigned to guest domain does not work.
Thank you for your reply. Your issue seems to be different from mine. Thanks, -- Yuji Shimada _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yuji Shimada
2008-Sep-03 08:35 UTC
Re: [Xen-devel] FC-HBA assigned to guest domain does not work.
(There was the typo. So I resend the mail.) Thank you for your reply. My patches say that some devices require that emulated BAR address is equal to actual BAR address. I don''t have internal design specification of FC-HBA. But I presume FC-HBA''s device driver and firmware communicate via its internal memory. And they use the same address to access internal memory. On non-virtualized environment, device driver and firmware use the same physical address. So they can communicate each other. On virtualized environment, device driver use guest physical address, while firmware use machine address. So they can''t communicate. My patches are not good solution to fix the issue. Hot-added device will not work, because Guest OS configures BARs. And other changes will be needed. For example, we need to change guest memory map to avoid conflict between MMIO resource and memory. Additionally guest physical address should be independent of machine address. I don''t have a good idea to solve the issue, currently. If VT-d provides some method to solve the issue, it is nice. -- Yuji Shimada On Tue, 02 Sep 2008 08:38:55 +0100 Keir Fraser <keir.fraser@eu.citrix.com> wrote:> The hvmloader side is perhaps plausible, since it only changes its behaviour > if qemu exposes an already-configured BAR. Fair enough. I don''t think that > blanket always-use-physical-BAR-address is a great idea in qemu -- perhaps > if it could be configurable that would be okay. > > -- Keir_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Apparently Analagous Threads
- Bug for Xen-3.3.0-rc2: libpci read error. No emulation.
- [PATCH][RFC] Support more Capability Structures and Device Specific
- [PATCH] fix memory allocation from NUMA node for VT-d.
- [PATCH] dom0 linux: Reassign memory resources to device for pci passthrough.
- [PATCH] Improve the current FLR logic