[libvirt-users] InfiniBand Passthrough not working

Michael Kruse
Tue, 12 Nov 2013 13:21:09 -0000

<div
style="font-family: Verdana;font-size:
12.0px;">
<span style="font-family:courier
new,courier,monospace;"> Hello,



<span style="font-family:courier
new,courier,monospace;">I want to passthrough an InfiniBand-Card to a
guest and have some problems with that. My CPU (Intel Xeon E5-2650) is vt-d
capable and vt-d is activated in the BIOS. My CentOS 6.4
(2.6.32-358.11.1.el6.x86_64) is booting with the command line &quot;...
intel_iommu=on iommu=pt".



<span style="font-family:courier
new,courier,monospace;">The guest has the same OS and kernel as the
host, also the driver used for the InfiniBand-Card is the same (used the
official Mellanox OFED driver pack).



<span style="font-family:courier
new,courier,monospace;">This is what I&#39;ve
done:



<span style="font-family:courier
new,courier,monospace;">0.) Verify that InfiniBand working on
host:


<span style="font-family:courier
new,courier,monospace;">root@host# ibv_devinfo
hca_id: mthca0
&nbsp;&nbsp; &nbsp;transport:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
InfiniBand (0)
&nbsp;&nbsp; &nbsp;fw_ver:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
1.0.800
&nbsp;&nbsp; &nbsp;node_guid:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;
...
&nbsp;&nbsp; &nbsp;sys_image_guid:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;
...
&nbsp;&nbsp; &nbsp;vendor_id:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
0x08f1
&nbsp;&nbsp; &nbsp;vendor_part_id:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
25204
&nbsp;&nbsp; &nbsp;hw_ver:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
0xA0
&nbsp;&nbsp; &nbsp;board_id:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;
...
&nbsp;&nbsp; &nbsp;phys_port_cnt:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
1
&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
port: 1
&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;state:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
PORT_ACTIVE (4)
&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;max_mtu:&nbsp;&nbsp;
2048 (4)
&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;active_mtu:&nbsp;&nbsp;
2048 (4)
&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;sm_lid:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
3
&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;port_lid:&nbsp;&nbsp;
5
&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;port_lmc:&nbsp;&nbsp;
0x00
&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;link_layer:&nbsp;&nbsp;
InfiniBand



<span style="font-family:courier
new,courier,monospace;">1.) DMAR and IOMMU messages at
host:


<span style="font-family:courier
new,courier,monospace;">root@host# dmesg&nbsp; &#124; grep -e
DMAR -e IOMMU
ACPI: DMAR 000000007e27ea30 00160 (v01 A M I&nbsp;&nbsp; OEMDMAR
00000001 INTL 00000001)
Intel-IOMMU: enabled
dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap
f020fe
dmar: IOMMU 1: reg_base_addr dfffc000 ver 1:0 cap d2078c106f0462 ecap
f020fe
IOMMU 0xfbffe000: using Queued invalidation
IOMMU 0xdfffc000: using Queued invalidation
IOMMU: hardware identity mapping for device
0000:00:00.0

<span style="font-family:courier
new,courier,monospace;">[... (a lot of mapping messages)
...]

<span style="font-family:courier
new,courier,monospace;">IOMMU: hardware identity mapping for device
0000:81:00.0 (this is the IB card)
IOMMU: Setting RMRR:
IOMMU: Prepare 0-16MiB unity mapping for LPC
dmar: DMAR:[DMA Read] Request device [81:00.0] fault addr 107294f000
DMAR:[fault reason 06] PTE Read access is not set
dmar: DMAR:[DMA Read] Request device [81:00.0] fault addr 107294f000
DMAR:[fault reason 06] PTE Read access is not set



<span style="font-family:courier
new,courier,monospace;">2.) Output of lspci at
host

<span style="font-family:courier
new,courier,monospace;">81:00.0 InfiniBand: Mellanox Technologies
MT25204 [InfiniHost III Lx HCA] (rev a0)
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Subsystem: Mellanox Technologies MT25204 [InfiniHost III Lx HCA]
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Flags:
fast devsel, IRQ 114
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Memory at
f8a00000 (64-bit, non-prefetchable) [size=1M]
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Memory at
3c1e00000000 (64-bit, prefetchable) [size=8M]
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Capabilities: [40] Power Management version 2
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Capabilities: [48] Vital Product Data
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Capabilities: [90] MSI: Enable- Count=1/32 Maskable- 64bit+
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Capabilities: [84] MSI-X: Enable- Count=32 Masked-
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Capabilities: [60] Express Endpoint, MSI 00

<span style="font-family:courier
new,courier,monospace;">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Kernel driver in use: ib_mthca
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Kernel
modules: ib_mthca



<span style="font-family:courier
new,courier,monospace;">3.) Adding the device with virsh
edit:
&nbsp;&nbsp;&nbsp; &lt;hostdev mode=&#39;subsystem&#39;
type=&#39;pci&#39; managed=&#39;yes&#39;>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
&lt;address domain=&#39;0x0000&#39; bus=&#39;0x81&#39;
slot=&#39;0x00&#39; function=&#39;0x0&#39;/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;address
type=&#39;pci&#39; domain=&#39;0x0000&#39;
bus=&#39;0x00&#39; slot=&#39;0x07&#39;
function=&#39;0x0&#39;/>
</hostdev>




<span style="font-family:courier
new,courier,monospace;">The last &lt;address .../&gt;-Tag with
&quot;slot=0x07&quot; was added automatically by
virsh.



<span style="font-family:courier
new,courier,monospace;">4.)&nbsp; Removing device with
nodedev-dettach:


<span style="font-family:courier
new,courier,monospace;">root@host# virsh nodedev-dettach
pci_0000_81_00_0
Device pci_0000_81_00_0 detached



<span style="font-family:courier
new,courier,monospace;">lspci on the host now shows the same as in (1)
but the &quot;Kernel driver in use&quot;
changed:


<span style="font-family:courier
new,courier,monospace;">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Kernel driver in use: pci-stub



<span style="font-family:courier
new,courier,monospace;">5.) InfiniBand now stops working on host (as
expected):


<span style="font-family:courier
new,courier,monospace;">root@host# ibv_devinfo
No IB devices found



<span style="font-family:courier
new,courier,monospace;">6.) starting guest, then&nbsp; lspci
-v:


<span style="font-family:courier
new,courier,monospace;">00:07.0 InfiniBand: Mellanox Technologies
MT25204 [InfiniHost III Lx HCA] (rev a0)
&nbsp;&nbsp; &nbsp;Subsystem: Mellanox Technologies MT25204
[InfiniHost III Lx HCA]
Physical Slot: 7
Flags: fast devsel, IRQ 10
&nbsp;&nbsp; &nbsp;Memory at f2100000 (32-bit, non-prefetchable)
[size=1M]
&nbsp;&nbsp; &nbsp;Memory at f2800000 (32-bit, prefetchable)
[size=8M]
Capabilities: [48] Vital Product Data
&nbsp;&nbsp; &nbsp;Capabilities: [60] Express Endpoint, MSI
00
&nbsp;&nbsp; &nbsp;Capabilities: [40] Power Management version
2
&nbsp;&nbsp; &nbsp;Capabilities: [84] MSI-X: Enable- Count=32
Masked-
&nbsp;&nbsp; &nbsp;Capabilities: [90] MSI: Enable- Count=1/32
Maskable- 64bit-
Kernel modules: ib_mthca



<span style="font-family:courier
new,courier,monospace;">I noticed that there is a difference: The memory
on guest is 32-bit, on the host it says 64-bit



<span style="font-family:courier
new,courier,monospace;">7.) IB not working on
guest:

<span style="font-family:courier
new,courier,monospace;">root@guest# ibv_devinfo
No IB devices found

<span style="font-family:courier
new,courier,monospace;">root@guest# ibhosts

<span style="font-family:courier
new,courier,monospace;">src/query_smp.c:228; can&#39;t open UMAD
port ((null):0)
/usr/sbin/ibnetdiscover: iberror: failed: discover
failed





<span style="font-family:courier
new,courier,monospace;">Do you have any clue where I must tweak my
settings in order to get InfiniBand working on my virtual machine? If this is no
libvirt problem, I am sorry :)



<span style="font-family:courier
new,courier,monospace;">Regards,

<span style="font-family:courier
new,courier,monospace;">Sebastian