Nadolski, Ed
2010-Mar-16 16:55 UTC
[Xen-devel] HVM hangs during boot with VTd passthru of QLogic FC HBA
I''m running Xen 4.0.0-rc6 with Fedora 12 dom0 on a Dell T7500 Xeon with
VT-x and VT-d enabled, and I have a Qlogic QLE2464 Fibre Channel HBA as a PCI
passthrough device. The Qlogic HBA works OK when I use it from dom0, but when I
try to assign it to an HVM with VT-d passthrough, the HVM fails to boot. I see
the Qlogic BIOS sign-on banner but then the HVM hangs, apparently somewhere
within the device BIOS code, and it never gets to the guest bootloader.
I''ve attached the console trace plus some other info below. The one
thing that looks suspicious to me is this:
(XEN) HVM2: PCI device 1077:2432 not found at index 4
(XEN) HVM2: PCI device 1033:014f not found at index 0
Any thoughts on what this might mean? Maybe some kind of mapping issue?
Thanks,
Ed
Here is the console trace, grub.conf, lspci, and HVM config file info:
(XEN) memory.c:132:d0 Could not allocate order=9 extent: id=2 memflags=0 (0 of
3)
(XEN) [VT-D]iommu.c:1435:d0 domain_context_unmap:PCIe: bdf = 24:0.0
(XEN) [VT-D]iommu.c:1309:d0 domain_context_mapping:PCIe: bdf = 24:0.0
(XEN) [VT-D]io.c:297:d0 VT-d irq bind: m_irq = 67 device = 4 intx = 0
(XEN) [VT-D]iommu.c:1435:d0 domain_context_unmap:PCIe: bdf = 24:0.1
(XEN) [VT-D]iommu.c:1309:d0 domain_context_mapping:PCIe: bdf = 24:0.1
(XEN) [VT-D]io.c:297:d0 VT-d irq bind: m_irq = 66 device = 5 intx = 1
(XEN) [VT-D]iommu.c:1435:d0 domain_context_unmap:PCIe: bdf = 25:0.0
(XEN) [VT-D]iommu.c:1309:d0 domain_context_mapping:PCIe: bdf = 25:0.0
(XEN) [VT-D]io.c:297:d0 VT-d irq bind: m_irq = 65 device = 6 intx = 0
(XEN) [VT-D]iommu.c:1435:d0 domain_context_unmap:PCIe: bdf = 25:0.1
(XEN) [VT-D]iommu.c:1309:d0 domain_context_mapping:PCIe: bdf = 25:0.1
(XEN) [VT-D]io.c:297:d0 VT-d irq bind: m_irq = 64 device = 7 intx = 1
(XEN) HVM2: HVM Loader
(XEN) HVM2: Detected Xen v4.0.0-rc6
(XEN) HVM2: CPU speed is 2128 MHz
(XEN) irq.c:243: Dom2 PCI link 0 changed 0 -> 5
(XEN) HVM2: PCI-ISA link 0 routed to IRQ5
(XEN) irq.c:243: Dom2 PCI link 1 changed 0 -> 10
(XEN) HVM2: PCI-ISA link 1 routed to IRQ10
(XEN) irq.c:243: Dom2 PCI link 2 changed 0 -> 11
(XEN) HVM2: PCI-ISA link 2 routed to IRQ11
(XEN) irq.c:243: Dom2 PCI link 3 changed 0 -> 5
(XEN) HVM2: PCI-ISA link 3 routed to IRQ5
(XEN) HVM2: pci dev 01:3 INTA->IRQ10
(XEN) HVM2: pci dev 03:0 INTA->IRQ5
(XEN) HVM2: pci dev 04:0 INTA->IRQ5
(XEN) HVM2: pci dev 05:0 INTB->IRQ11
(XEN) HVM2: pci dev 06:0 INTA->IRQ11
(XEN) HVM2: pci dev 07:0 INTB->IRQ5
(XEN) HVM2: pci dev 02:0 bar 10 size 02000000: f0000008
(XEN) HVM2: pci dev 03:0 bar 14 size 01000000: f2000008
(XEN) HVM2: pci dev 04:0 bar 30 size 00040000: f3000000
(XEN) HVM2: pci dev 05:0 bar 30 size 00040000: f3040000
(XEN) HVM2: pci dev 06:0 bar 30 size 00040000: f3080000
(XEN) HVM2: pci dev 07:0 bar 30 size 00040000: f30c0000
(XEN) domctl.c:991:d0 memory_map:add: gfn=f3100 mfn=dfbf8 nr_mfns=4
(XEN) domctl.c:1001:d0 memory_map:remove: gfn=f3102 mfn=dfbfa nr_mfns=1
(XEN) HVM2: pci dev 04:0 bar 14 size 00004000: f3100004
(XEN) domctl.c:991:d0 memory_map:add: gfn=f3104 mfn=dfbfc nr_mfns=4
(XEN) domctl.c:1001:d0 memory_map:remove: gfn=f3106 mfn=dfbfe nr_mfns=1
(XEN) HVM2: pci dev 05:0 bar 14 size 00004000: f3104004
(XEN) domctl.c:991:d0 memory_map:add: gfn=f3108 mfn=df9f8 nr_mfns=4
(XEN) domctl.c:1001:d0 memory_map:remove: gfn=f310a mfn=df9fa nr_mfns=1
(XEN) HVM2: pci dev 06:0 bar 14 size 00004000: f3108004
(XEN) domctl.c:991:d0 memory_map:add: gfn=f310c mfn=df9fc nr_mfns=4
(XEN) domctl.c:1001:d0 memory_map:remove: gfn=f310e mfn=df9fe nr_mfns=1
(XEN) HVM2: pci dev 07:0 bar 14 size 00004000: f310c004
(XEN) HVM2: pci dev 02:0 bar 14 size 00001000: f3110000
(XEN) HVM2: pci dev 03:0 bar 10 size 00000100: 0000c001
(XEN) HVM2: pci dev 04:0 bar 10 size 00000100: 0000c101
(XEN) domctl.c:1047:d0 ioport_map:add f_gport=c100 f_mport=e800 np=100
(XEN) HVM2: pci dev 05:0 bar 10 size 00000100: 0000c201
(XEN) domctl.c:1047:d0 ioport_map:add f_gport=c200 f_mport=ec00 np=100
(XEN) HVM2: pci dev 06:0 bar 10 size 00000100: 0000c301
(XEN) domctl.c:1047:d0 ioport_map:add f_gport=c300 f_mport=d800 np=100
(XEN) HVM2: pci dev 07:0 bar 10 size 00000100: 0000c401
(XEN) domctl.c:1047:d0 ioport_map:add f_gport=c400 f_mport=dc00 np=100
(XEN) HVM2: pci dev 01:1 bar 20 size 00000010: 0000c501
(XEN) HVM2: Multiprocessor initialisation:
(XEN) HVM2: - CPU0 ... 40-bit phys ... fixed MTRRs ... var MTRRs [2/8] ...
done.
(XEN) HVM2: Writing SMBIOS tables ...
(XEN) HVM2: Loading ROMBIOS ...
(XEN) HVM2: 10524 bytes of ROMBIOS high-memory extensions:
(XEN) HVM2: Relocating to 0xfc000000-0xfc00291c ... done
(XEN) HVM2: Creating MP tables ...
(XEN) HVM2: Loading Cirrus VGABIOS ...
(XEN) domctl.c:991:d0 memory_map:add: gfn=f3000 mfn=dfc00 nr_mfns=40
(XEN) HVM2: Loading PCI Option ROM ...
(XEN) HVM2: - Manufacturer: QLogic
(XEN) HVM2: - Product name:
(XEN) domctl.c:1001:d0 memory_map:remove: gfn=f3000 mfn=dfc00 nr_mfns=40
(XEN) domctl.c:991:d0 memory_map:add: gfn=f3040 mfn=dc000 nr_mfns=40
(XEN) domctl.c:1001:d0 memory_map:remove: gfn=f3040 mfn=dc000 nr_mfns=40
(XEN) domctl.c:991:d0 memory_map:add: gfn=f3080 mfn=dfa00 nr_mfns=40
(XEN) domctl.c:1001:d0 memory_map:remove: gfn=f3080 mfn=dfa00 nr_mfns=40
(XEN) domctl.c:991:d0 memory_map:add: gfn=f30c0 mfn=dc100 nr_mfns=40
(XEN) domctl.c:1001:d0 memory_map:remove: gfn=f30c0 mfn=dc100 nr_mfns=40
(XEN) HVM2: Loading ACPI ...
(XEN) HVM2: - Lo data: 000ea020-000ea04f
(XEN) HVM2: - Hi data: fc002c00-fc012d1f
(XEN) HVM2: vm86 TSS at fc013000
(XEN) HVM2: BIOS map:
(XEN) HVM2: c0000-c8fff: VGA BIOS
(XEN) HVM2: c9000-d47ff: PCI Option ROMs
(XEN) HVM2: eb000-eb148: SMBIOS tables
(XEN) HVM2: f0000-fffff: Main BIOS
(XEN) HVM2: Invoking ROMBIOS ...
(XEN) HVM2: $Revision: 1.221 $ $Date: 2008/12/07 17:32:29 $
(XEN) stdvga.c:147:d2 entering stdvga and caching modes
(XEN) HVM2: VGABios $Id: vgabios.c,v 1.67 2008/01/27 09:44:12 vruppert Exp $
(XEN) HVM2: Bochs BIOS - build: 06/23/99
(XEN) HVM2: $Revision: 1.221 $ $Date: 2008/12/07 17:32:29 $
(XEN) HVM2: Options: apmbios pcibios eltorito PMM
(XEN) HVM2:
(XEN) HVM2: ata0-0: PCHS=16383/16/63 translation=lba LCHS=1024/255/63
(XEN) HVM2: ata0 master: QEMU HARDDISK ATA-7 Hard-Disk (20480 MBytes)
(XEN) HVM2: IDE time out
(XEN) HVM2:
(XEN) HVM2: PCI device 1077:2432 not found at index 4
(XEN) HVM2: PCI device 1033:014f not found at index 0
In /etc/grub.conf:
title Fedora-12 Xen 4.0.0-rc6 (2.6.31.12) iommu=1
xen-pciback.hide=(24:00.0)(24:00.1)(25:00.0)(25:00.1)
root (hd0,0)
kernel /xen-4.0.0-rc6.gz iommu=1 acpi_skip_timer_override loglvl=all
guest_loglvl=all sync_console console_to_ring com1=115200,8n1 console=com1
module /vmlinuz-2.6.31.12 ro
root=UUID=edbcbc29-f3e4-4985-80c1-3c3b0ce24d17 LANG=en_US.UTF-8
SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us console=hvc0
earlyprintk=xen xen-pciback.hide=(24:00.0)(24:00.1)(25:00.0)(25:00.1)
module /initramfs-2.6.31.12.img
In /etc/xen/xend-config.sxp:
(pci-passthrough-strict-check no)
# xm pci-list-assignable-devices
0000:25:00.0
0000:25:00.1
0000:24:00.0
0000:24:00.1
# lspci | grep -i ql
24:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI
Express HBA (rev 03)
24:00.1 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI
Express HBA (rev 03)
25:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI
Express HBA (rev 03)
25:00.1 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI
Express HBA (rev 03)
# lspci -vvvxxx -s 24:00.0
24:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI
Express HBA (rev 03)
Subsystem: QLogic Corp. Device 0146
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 61
Region 0: I/O ports at e800 [disabled] [size=256]
Region 1: Memory at dfbf8000 (64-bit, non-prefetchable) [disabled]
[size=16K]
Expansion ROM at dfc00000 [disabled] [size=256K]
Capabilities: [44] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [4c] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <4us,
L1 <1us
ExtTag- AttnBtn+ AttnInd+ PwrInd+ RBE- FLReset-
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr-
TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s, Latency L0
<4us, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
Capabilities: [64] MSI: Enable- Count=1/16 Maskable- 64bit+
Address: 00000000fee0800c Data: 412c
Capabilities: [74] Vital Product Data
Product Name: PCI-Express Quad Channel 4Gb Fibre Channel HBA
Read-only fields:
[PN] Part number: QLE2464
[SN] Serial number: GFC0709T71961
[V0] Vendor specific: PW=15W
[MN] Manufacture ID: 50 58 32 36 31 30 34 30 32 2d 30 31
20 30 36
[V1] Vendor specific: 01.05
[V3] Vendor specific: 04.00.22
[V4] Vendor specific: 01.18
[V5] Vendor specific: 01.17
[YA] Asset tag:
[RV] Reserved: checksum good, 0 byte(s) reserved
End
Capabilities: [7c] MSI-X: Enable- Count=16 Masked-
Vector table: BAR=1 offset=00002000
PBA: BAR=1 offset=00003000
Capabilities: [100] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF-
MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+
MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [138] Power Budgeting <?>
Kernel driver in use: pciback
Kernel modules: qla2xxx
00: 77 10 32 24 00 00 10 00 03 00 04 0c 10 00 80 00
10: 01 e8 00 00 04 80 bf df 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 77 10 46 01
30: 00 00 c0 df 44 00 00 00 00 00 00 00 0a 01 00 00
40: 00 00 00 00 01 4c 02 00 00 00 00 00 10 64 01 00
50: 83 71 64 00 36 28 0a 00 41 e4 03 00 40 00 41 10
60: 00 00 00 00 05 74 88 00 0c 80 e0 fe 00 00 00 00
70: 2c 41 00 00 03 7c a0 80 56 01 28 78 11 00 0f 00
80: 01 20 00 00 01 30 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
HVM config file:
# -*- mode: python; -*-
kernel = "/usr/lib/xen/boot/hvmloader"
builder=''hvm''
memory = 512
name = "f10hvm64"
device_model = ''/usr/lib/xen/bin/qemu-dm''
disk = [ ''file:/root/Desktop/vm_files/f10hvm64.img,hda,w'' ]
boot="c"
sdl=0
vnc=1
vncpasswd=''''
stdvga=0
serial=''pty''
pci = [ ''24:00.0'', ''24:00.1'',
''25:00.0'', ''25:00.1'' ]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Mar-16 17:38 UTC
Re: [Xen-devel] HVM hangs during boot with VTd passthru of QLogic FC HBA
> (XEN) HVM2: PCI device 1033:014f not found at index 0Which second device is that?> > Any thoughts on what this might mean? Maybe some kind of mapping issue?.. snip..> (XEN) domctl.c:991:d0 memory_map:add: gfn=f3000 mfn=dfc00 nr_mfns=40Here the BIOS is loaded...> (XEN) HVM2: Loading PCI Option ROM ... > (XEN) HVM2: - Manufacturer: QLogic > (XEN) HVM2: - Product name: > (XEN) domctl.c:1001:d0 memory_map:remove: gfn=f3000 mfn=dfc00 nr_mfns=40Unloaded..> (XEN) domctl.c:991:d0 memory_map:add: gfn=f3040 mfn=dc000 nr_mfns=40Then something at a lower location (DC00) gets loaded> (XEN) domctl.c:1001:d0 memory_map:remove: gfn=f3040 mfn=dc000 nr_mfns=40unloaded..> (XEN) domctl.c:991:d0 memory_map:add: gfn=f3080 mfn=dfa00 nr_mfns=40 > (XEN) domctl.c:1001:d0 memory_map:remove: gfn=f3080 mfn=dfa00 nr_mfns=40 > (XEN) domctl.c:991:d0 memory_map:add: gfn=f30c0 mfn=dc100 nr_mfns=40 > (XEN) domctl.c:1001:d0 memory_map:remove: gfn=f30c0 mfn=dc100 nr_mfns=40That looks to me that it actually unloaded the QLogic BIOS? I would suggest you instrument QEMU/Bochs BIOS (or turn the debugging way up)to see how it loads the BIOS. It could be that it cannot find space for a 256kB BIOS and tries its best.> (XEN) HVM2: Loading ACPI ... > (XEN) HVM2: - Lo data: 000ea020-000ea04f > (XEN) HVM2: - Hi data: fc002c00-fc012d1f > (XEN) HVM2: vm86 TSS at fc013000 > (XEN) HVM2: BIOS map: > (XEN) HVM2: c0000-c8fff: VGA BIOS > (XEN) HVM2: c9000-d47ff: PCI Option ROMs > (XEN) HVM2: eb000-eb148: SMBIOS tables > (XEN) HVM2: f0000-fffff: Main BIOS > (XEN) HVM2: Invoking ROMBIOS ... > (XEN) HVM2: $Revision: 1.221 $ $Date: 2008/12/07 17:32:29 $ > (XEN) stdvga.c:147:d2 entering stdvga and caching modes > (XEN) HVM2: VGABios $Id: vgabios.c,v 1.67 2008/01/27 09:44:12 vruppert Exp $ > (XEN) HVM2: Bochs BIOS - build: 06/23/99 > (XEN) HVM2: $Revision: 1.221 $ $Date: 2008/12/07 17:32:29 $ > (XEN) HVM2: Options: apmbios pcibios eltorito PMM > (XEN) HVM2: > (XEN) HVM2: ata0-0: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 > (XEN) HVM2: ata0 master: QEMU HARDDISK ATA-7 Hard-Disk (20480 MBytes) > (XEN) HVM2: IDE time out > (XEN) HVM2: > (XEN) HVM2: PCI device 1077:2432 not found at index 4 > (XEN) HVM2: PCI device 1033:014f not found at index 0_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel