thr3ads.net - libvirt users - [libvirt-users] InfiniBand Passthrough not working [Nov 2013]

If this information is useful, please help other people find it:
Share via:
Michael Kruse
2013-Nov-12 13:21 UTC
[libvirt-users] InfiniBand Passthrough not working

<html><head></head><body><div
style="font-family: Verdana;font-size:
12.0px;"><div>&nbsp;
<div><span style="font-family:courier
new,courier,monospace;">&nbsp;Hello,</span>

<div>&nbsp;</div>

<div><span style="font-family:courier
new,courier,monospace;">I want to passthrough an InfiniBand-Card to a
guest and have some problems with that. My CPU (Intel Xeon E5-2650) is vt-d
capable and vt-d is activated in the BIOS. My CentOS 6.4
(2.6.32-358.11.1.el6.x86_64) is booting with the command line &quot;...
intel_iommu=on iommu=pt&quot;. </span></div>

<div>&nbsp;</div>

<div><span style="font-family:courier
new,courier,monospace;">The guest has the same OS and kernel as the
host, also the driver used for the InfiniBand-Card is the same (used the
official Mellanox OFED driver pack).</span></div>

<div>&nbsp;</div>

<div><span style="font-family:courier
new,courier,monospace;">This is what I&#39;ve
done:</span></div>

<div>&nbsp;</div>

<div><span style="font-family:courier
new,courier,monospace;">0.) Verify that InfiniBand working on
host:</span></div>

<div>
<div><span style="font-family:courier
new,courier,monospace;">root@host# ibv_devinfo<br/>
hca_id:&nbsp;&nbsp; &nbsp;mthca0<br/>
&nbsp;&nbsp; &nbsp;transport:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;InfiniBand (0)<br/>
&nbsp;&nbsp; &nbsp;fw_ver:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;1.0.800<br/>
&nbsp;&nbsp; &nbsp;node_guid:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;
...<br/>
&nbsp;&nbsp; &nbsp;sys_image_guid:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;
...<br/>
&nbsp;&nbsp; &nbsp;vendor_id:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;0x08f1<br/>
&nbsp;&nbsp; &nbsp;vendor_part_id:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;25204<br/>
&nbsp;&nbsp; &nbsp;hw_ver:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;0xA0<br/>
&nbsp;&nbsp; &nbsp;board_id:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;
...<br/>
&nbsp;&nbsp; &nbsp;phys_port_cnt:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;1<br/>
&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;port:&nbsp;&nbsp; &nbsp;1<br/>
&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;state:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;PORT_ACTIVE (4)<br/>
&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;max_mtu:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;2048 (4)<br/>
&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;active_mtu:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;2048 (4)<br/>
&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;sm_lid:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;3<br/>
&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;port_lid:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;5<br/>
&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;port_lmc:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;0x00<br/>
&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;link_layer:&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp; &nbsp;InfiniBand</span></div>

<div>&nbsp;</div>

<div><span style="font-family:courier
new,courier,monospace;">1.) DMAR and IOMMU messages at
host:</span></div>

<div>
<div><span style="font-family:courier
new,courier,monospace;">root@host# dmesg&nbsp; &#124; grep -e
DMAR -e IOMMU<br/>
ACPI: DMAR 000000007e27ea30 00160 (v01 A M I&nbsp;&nbsp; OEMDMAR
00000001 INTL 00000001)<br/>
Intel-IOMMU: enabled<br/>
dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap
f020fe<br/>
dmar: IOMMU 1: reg_base_addr dfffc000 ver 1:0 cap d2078c106f0462 ecap
f020fe<br/>
IOMMU 0xfbffe000: using Queued invalidation<br/>
IOMMU 0xdfffc000: using Queued invalidation<br/>
IOMMU: hardware identity mapping for device
0000:00:00.0</span></div>

<div><span style="font-family:courier
new,courier,monospace;">[... (a lot of mapping messages)
...]</span></div>

<div><span style="font-family:courier
new,courier,monospace;">IOMMU: hardware identity mapping for device
0000:81:00.0 (this is the IB card)<br/>
IOMMU: Setting RMRR:<br/>
IOMMU: Prepare 0-16MiB unity mapping for LPC<br/>
dmar: DMAR:[DMA Read] Request device [81:00.0] fault addr 107294f000<br/>
DMAR:[fault reason 06] PTE Read access is not set<br/>
dmar: DMAR:[DMA Read] Request device [81:00.0] fault addr 107294f000<br/>
DMAR:[fault reason 06] PTE Read access is not set</span></div>
</div>
</div>

<div><span style="font-family:courier
new,courier,monospace;">2.) Output of lspci at
host</span></div>

<div><span style="font-family:courier
new,courier,monospace;">81:00.0 InfiniBand: Mellanox Technologies
MT25204 [InfiniHost III Lx HCA] (rev a0)<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Subsystem: Mellanox Technologies MT25204 [InfiniHost III Lx HCA]<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Flags:
fast devsel, IRQ 114<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Memory at
f8a00000 (64-bit, non-prefetchable) [size=1M]<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Memory at
3c1e00000000 (64-bit, prefetchable) [size=8M]<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Capabilities: [40] Power Management version 2<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Capabilities: [48] Vital Product Data<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Capabilities: [90] MSI: Enable- Count=1/32 Maskable- 64bit+<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Capabilities: [84] MSI-X: Enable- Count=32 Masked-<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Capabilities: [60] Express Endpoint, MSI 00</span></div>

<div><span style="font-family:courier
new,courier,monospace;">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Kernel driver in use: ib_mthca<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Kernel
modules: ib_mthca</span></div>

<div>&nbsp;</div>

<div><span style="font-family:courier
new,courier,monospace;">3.) Adding the device with virsh
edit:<br/>
&nbsp;&nbsp;&nbsp; &lt;hostdev mode=&#39;subsystem&#39;
type=&#39;pci&#39; managed=&#39;yes&#39;&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
&lt;source&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
&lt;address domain=&#39;0x0000&#39; bus=&#39;0x81&#39;
slot=&#39;0x00&#39; function=&#39;0x0&#39;/&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
&lt;/source&gt;<br/>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;address
type=&#39;pci&#39; domain=&#39;0x0000&#39;
bus=&#39;0x00&#39; slot=&#39;0x07&#39;
function=&#39;0x0&#39;/&gt;<br/>
&nbsp;&nbsp;&nbsp; &lt;/hostdev&gt;</span></div>

<div>
<div>&nbsp;</div>

<div><span style="font-family:courier
new,courier,monospace;">The last &lt;address .../&gt;-Tag with
&quot;slot=0x07&quot; was added automatically by
virsh.</span></div>

<div>&nbsp;</div>

<div><span style="font-family:courier
new,courier,monospace;">4.)&nbsp; Removing device with
nodedev-dettach:</span></div>

<div>
<div><span style="font-family:courier
new,courier,monospace;">root@host# virsh nodedev-dettach
pci_0000_81_00_0<br/>
Device pci_0000_81_00_0 detached</span></div>

<div>&nbsp;</div>

<div><span style="font-family:courier
new,courier,monospace;">lspci on the host now shows the same as in (1)
but the &quot;Kernel driver in use&quot;
changed:</span></div>
</div>

<div><span style="font-family:courier
new,courier,monospace;">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
Kernel driver in use: pci-stub</span></div>

<div>&nbsp;</div>

<div><span style="font-family:courier
new,courier,monospace;">5.) InfiniBand now stops working on host (as
expected):</span></div>

<div>
<div><span style="font-family:courier
new,courier,monospace;">root@host# ibv_devinfo<br/>
No IB devices found</span></div>

<div>&nbsp;</div>

<div><span style="font-family:courier
new,courier,monospace;">6.) starting guest, then&nbsp; lspci
-v:</span></div>

<div>
<div><span style="font-family:courier
new,courier,monospace;">00:07.0 InfiniBand: Mellanox Technologies
MT25204 [InfiniHost III Lx HCA] (rev a0)<br/>
&nbsp;&nbsp; &nbsp;Subsystem: Mellanox Technologies MT25204
[InfiniHost III Lx HCA]<br/>
&nbsp;&nbsp; &nbsp;Physical Slot: 7<br/>
&nbsp;&nbsp; &nbsp;Flags: fast devsel, IRQ 10<br/>
&nbsp;&nbsp; &nbsp;Memory at f2100000 (32-bit, non-prefetchable)
[size=1M]<br/>
&nbsp;&nbsp; &nbsp;Memory at f2800000 (32-bit, prefetchable)
[size=8M]<br/>
&nbsp;&nbsp; &nbsp;Capabilities: [48] Vital Product Data<br/>
&nbsp;&nbsp; &nbsp;Capabilities: [60] Express Endpoint, MSI
00<br/>
&nbsp;&nbsp; &nbsp;Capabilities: [40] Power Management version
2<br/>
&nbsp;&nbsp; &nbsp;Capabilities: [84] MSI-X: Enable- Count=32
Masked-<br/>
&nbsp;&nbsp; &nbsp;Capabilities: [90] MSI: Enable- Count=1/32
Maskable- 64bit-<br/>
&nbsp;&nbsp; &nbsp;Kernel modules: ib_mthca</span></div>

<div>&nbsp;</div>

<div><span style="font-family:courier
new,courier,monospace;">I noticed that there is a difference: The memory
on guest is 32-bit, on the host it says 64-bit</span></div>

<div>&nbsp;</div>

<div><span style="font-family:courier
new,courier,monospace;">7.) IB not working on
guest:</span></div>

<div><span style="font-family:courier
new,courier,monospace;">root@guest# ibv_devinfo<br/>
No IB devices found</span></div>

<div><span style="font-family:courier
new,courier,monospace;">root@guest# ibhosts</span></div>

<div><span style="font-family:courier
new,courier,monospace;">src/query_smp.c:228; can&#39;t open UMAD
port ((null):0)<br/>
/usr/sbin/ibnetdiscover: iberror: failed: discover
failed</span></div>
</div>
</div>

<div>&nbsp;</div>

<div><span style="font-family:courier
new,courier,monospace;">Do you have any clue where I must tweak my
settings in order to get InfiniBand working on my virtual machine? If this is no
libvirt problem, I am sorry :)</span></div>

<div>&nbsp;</div>

<div><span style="font-family:courier
new,courier,monospace;">Regards,</span></div>

<div><span style="font-family:courier
new,courier,monospace;">Sebastian</span></div>
</div>
</div>
</div></div></body></html>
Reasonably Related Threads

Search for more reasonably related threads
libvirt users - Nov 2013 - InfiniBand Passthrough not working

[libvirt-users] InfiniBand Passthrough not working

Reasonably Related Threads

Wisdom of the Ancients
