Yalan Zhang
2018-Jul-03 13:37 UTC
[libvirt-users] multiple devices in the same iommu group in L1 guest
Hi,
I have a guest enabled vIOMMU, but on the guest there are several devices
in the same iommu group.
Could someone help to check if I missed something?
Thank you very much!
1. guest xml:
# virsh edit q
...
<os>
<type arch='x86_64'
machine='pc-q35-rhel7.5.0'>hvm</type>
<loader readonly='yes' secure='yes'
type='pflash'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/q_VARS.fd</nvram>
</os>
...
<features>
...
<ioapic driver='qemu'/>
</features>
<cpu mode='host-passthrough' check='none'>
<feature policy='require' name='vmx'/>
</cpu>
...
<devices>
...
<controller type='pci' index='7'
model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='7' port='0x15'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x5'/>
</controller>
<interface type='network'>
<mac address='52:54:00:b9:ff:90'/>
<source network='default'/>
<model type='e1000'/>
<address type='pci' domain='0x0000' bus='0x07'
slot='0x00'
function='0x0'/>
</interface>
<iommu model='intel'>
<driver intremap='on' caching_mode='on'
iotlb='on'/>
</iommu>
</devices>
...
2. guest has 'intel_iommu=on' enabled in kernel cmdline, then reboot
guest
3. log in guest to check:
# dmesg | grep -i DMAR
[ 0.000000] ACPI: DMAR 000000007d83f000 00050 (v01 BOCHS BXPCDMAR
00000001 BXPC 00000001)
[ 0.000000] DMAR: IOMMU enabled
[ 0.155178] DMAR: Host address width 39
[ 0.155180] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[ 0.155221] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap
12008c22260286 ecap f00f5e
[ 0.155228] DMAR: ATSR flags: 0x1
[ 0.155231] DMAR-IR: IOAPIC id 0 under DRHD base 0xfed90000 IOMMU 0
[ 0.155232] DMAR-IR: Queued invalidation will be enabled to support
x2apic and Intr-remapping.
[ 0.156843] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 2.112369] DMAR: No RMRR found
[ 2.112505] DMAR: dmar0: Using Queued invalidation
[ 2.112669] DMAR: Setting RMRR:
[ 2.112671] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 2.112820] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 -
0xffffff]
[ 2.211577] DMAR: Intel(R) Virtualization Technology for Directed I/O
===> This is expected
# dmesg | grep -i iommu |grep device
[ 2.212267] iommu: Adding device 0000:00:00.0 to group 0
[ 2.212287] iommu: Adding device 0000:00:01.0 to group 1
[ 2.212372] iommu: Adding device 0000:00:02.0 to group 2
[ 2.212392] iommu: Adding device 0000:00:02.1 to group 2
[ 2.212411] iommu: Adding device 0000:00:02.2 to group 2
[ 2.212444] iommu: Adding device 0000:00:02.3 to group 2
[ 2.212464] iommu: Adding device 0000:00:02.4 to group 2
[ 2.212482] iommu: Adding device 0000:00:02.5 to group 2
[ 2.212520] iommu: Adding device 0000:00:1d.0 to group 3
[ 2.212533] iommu: Adding device 0000:00:1d.1 to group 3
[ 2.212541] iommu: Adding device 0000:00:1d.2 to group 3
[ 2.212550] iommu: Adding device 0000:00:1d.7 to group 3
[ 2.212567] iommu: Adding device 0000:00:1f.0 to group 4
[ 2.212576] iommu: Adding device 0000:00:1f.2 to group 4
[ 2.212585] iommu: Adding device 0000:00:1f.3 to group 4
[ 2.212599] iommu: Adding device 0000:01:00.0 to group 2
[ 2.212605] iommu: Adding device 0000:02:01.0 to group 2
[ 2.212621] iommu: Adding device 0000:04:00.0 to group 2
[ 2.212634] iommu: Adding device 0000:05:00.0 to group 2
[ 2.212646] iommu: Adding device 0000:06:00.0 to group 2
[ 2.212657] iommu: Adding device 0000:07:00.0 to group 2
====> several devices in the same iommu group
# virsh nodedev-dumpxml pci_0000_07_00_0
<device>
<name>pci_0000_07_00_0</name>
<path>/sys/devices/pci0000:00/0000:00:02.5/0000:07:00.0</path>
<parent>pci_0000_00_02_5</parent>
<driver>
<name>e1000</name>
</driver>
<capability type='pci'>
<domain>0</domain>
<bus>7</bus>
<slot>0</slot>
<function>0</function>
<product id='0x100e'>82540EM Gigabit Ethernet
Controller</product>
<vendor id='0x8086'>Intel Corporation</vendor>
<iommuGroup number='2'>
<address domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
<address domain='0x0000' bus='0x00' slot='0x02'
function='0x1'/>
<address domain='0x0000' bus='0x00' slot='0x02'
function='0x2'/>
<address domain='0x0000' bus='0x00' slot='0x02'
function='0x3'/>
<address domain='0x0000' bus='0x00' slot='0x02'
function='0x4'/>
<address domain='0x0000' bus='0x00' slot='0x02'
function='0x5'/>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
<address domain='0x0000' bus='0x02' slot='0x01'
function='0x0'/>
<address domain='0x0000' bus='0x04' slot='0x00'
function='0x0'/>
<address domain='0x0000' bus='0x05' slot='0x00'
function='0x0'/>
<address domain='0x0000' bus='0x06' slot='0x00'
function='0x0'/>
<address domain='0x0000' bus='0x07' slot='0x00'
function='0x0'/>
</iommuGroup>
</capability>
</device>
Thus, can not attach the device to L2 guest:
# cat hostdev.xml
<hostdev mode='subsystem' type='pci'
managed='yes'>
<source>
<address domain='0x0000' bus='0x07' slot='0x00'
function='0x0'/>
</source>
</hostdev>
# virsh attach-device rhel hostdev.xml
error: Failed to attach device from hostdev.xml
error: internal error: unable to execute QEMU command 'device_add': vfio
error: 0000:07:00.0: group 2 is not viable
-------
Best Regards,
Yalan Zhang
IRC: yalzhang
Internal phone: 8389413
Laine Stump
2018-Aug-06 19:28 UTC
Re: [libvirt-users] multiple devices in the same iommu group in L1 guest
On 07/03/2018 09:37 AM, Yalan Zhang wrote:> Hi, > > I have a guest enabled vIOMMU, but on the guest there are several > devices in the same iommu group. > Could someone help to check if I missed something? > Thank you very much! > > 1. guest xml: > # virsh edit q > ... > <os> > <type arch='x86_64' machine='pc-q35-rhel7.5.0'>hvm</type> > <loader readonly='yes' secure='yes' > type='pflash'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader> > <nvram>/var/lib/libvirt/qemu/nvram/q_VARS.fd</nvram> > </os> > ... > <features> > ... > <ioapic driver='qemu'/> > </features> > <cpu mode='host-passthrough' check='none'> > <feature policy='require' name='vmx'/> > </cpu> > ... > <devices> > ... > <controller type='pci' index='7' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='7' port='0x15'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > function='0x5'/> > </controller> > <interface type='network'> > <mac address='52:54:00:b9:ff:90'/> > <source network='default'/> > <model type='e1000'/> > <address type='pci' domain='0x0000' bus='0x07' slot='0x00' > function='0x0'/> > </interface> > <iommu model='intel'> > <driver intremap='on' caching_mode='on' iotlb='on'/> > </iommu> > </devices> > ... > 2. guest has 'intel_iommu=on' enabled in kernel cmdline, then reboot guest > > 3. log in guest to check: > # dmesg | grep -i DMAR > [ 0.000000] ACPI: DMAR 000000007d83f000 00050 (v01 BOCHS BXPCDMAR > 00000001 BXPC 00000001) > [ 0.000000] DMAR: IOMMU enabled > [ 0.155178] DMAR: Host address width 39 > [ 0.155180] DMAR: DRHD base: 0x000000fed90000 flags: 0x1 > [ 0.155221] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap > 12008c22260286 ecap f00f5e > [ 0.155228] DMAR: ATSR flags: 0x1 > [ 0.155231] DMAR-IR: IOAPIC id 0 under DRHD base 0xfed90000 IOMMU 0 > [ 0.155232] DMAR-IR: Queued invalidation will be enabled to support > x2apic and Intr-remapping. > [ 0.156843] DMAR-IR: Enabled IRQ remapping in x2apic mode > [ 2.112369] DMAR: No RMRR found > [ 2.112505] DMAR: dmar0: Using Queued invalidation > [ 2.112669] DMAR: Setting RMRR: > [ 2.112671] DMAR: Prepare 0-16MiB unity mapping for LPC > [ 2.112820] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - > 0xffffff] > [ 2.211577] DMAR: Intel(R) Virtualization Technology for Directed I/O > ===> This is expected > > # dmesg | grep -i iommu |grep device > [ 2.212267] iommu: Adding device 0000:00:00.0 to group 0 > [ 2.212287] iommu: Adding device 0000:00:01.0 to group 1 > [ 2.212372] iommu: Adding device 0000:00:02.0 to group 2 > [ 2.212392] iommu: Adding device 0000:00:02.1 to group 2 > [ 2.212411] iommu: Adding device 0000:00:02.2 to group 2 > [ 2.212444] iommu: Adding device 0000:00:02.3 to group 2 > [ 2.212464] iommu: Adding device 0000:00:02.4 to group 2 > [ 2.212482] iommu: Adding device 0000:00:02.5 to group 2 > [ 2.212520] iommu: Adding device 0000:00:1d.0 to group 3 > [ 2.212533] iommu: Adding device 0000:00:1d.1 to group 3 > [ 2.212541] iommu: Adding device 0000:00:1d.2 to group 3 > [ 2.212550] iommu: Adding device 0000:00:1d.7 to group 3 > [ 2.212567] iommu: Adding device 0000:00:1f.0 to group 4 > [ 2.212576] iommu: Adding device 0000:00:1f.2 to group 4 > [ 2.212585] iommu: Adding device 0000:00:1f.3 to group 4 > [ 2.212599] iommu: Adding device 0000:01:00.0 to group 2 > [ 2.212605] iommu: Adding device 0000:02:01.0 to group 2 > [ 2.212621] iommu: Adding device 0000:04:00.0 to group 2 > [ 2.212634] iommu: Adding device 0000:05:00.0 to group 2 > [ 2.212646] iommu: Adding device 0000:06:00.0 to group 2 > [ 2.212657] iommu: Adding device 0000:07:00.0 to group 2 > ====> several devices in the same iommu group > > # virsh nodedev-dumpxml pci_0000_07_00_0 > <device> > <name>pci_0000_07_00_0</name> > <path>/sys/devices/pci0000:00/0000:00:02.5/0000:07:00.0</path> > <parent>pci_0000_00_02_5</parent> > <driver> > <name>e1000</name> > </driver> > <capability type='pci'> > <domain>0</domain> > <bus>7</bus> > <slot>0</slot> > <function>0</function> > <product id='0x100e'>82540EM Gigabit Ethernet Controller</product> > <vendor id='0x8086'>Intel Corporation</vendor> > <iommuGroup number='2'> > <address domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> > <address domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> > <address domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> > <address domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> > <address domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> > <address domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> > <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> > <address domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> > <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> > <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> > <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> > <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> > </iommuGroup>All devices plugged into the same slot on pcie-root are in the same iommu group, and all devices are in the same iommu group as the pcie-root-port they are plugged into. Since libvirt groups 8 pcie-root-ports onto a single pcie-root slot (to conserve the relatively limited number of slots), all the devices plugged into those 8 pcie-root-ports are in the same iommu group. This can be solved by putting the pcie-root-ports *of the devices you want to assign to L2 guests* onto pcie-root slots by themselves, i.e. instead of addressing it at 00:02.5, address it at 00:10.0 (and don't put any other pcie-root-ports at 00:10.x). I had thought there was an open BZ about this, but can't find one anywhere. At first glance it would seem that, since the pcie-root-ports and iommu controller are all emulated devices, it would be simple to just report all the pcie-root-ports as being on different iommu groups regardless of their placement on pcie-root, but I believe when this was brought up doubt was expressed over whether or not qemu truly can guarantee that the devices are fully isolated from each other.> </capability> > </device> > > Thus, can not attach the device to L2 guest: > # cat hostdev.xml > <hostdev mode='subsystem' type='pci' managed='yes'> > <source> > <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> > </source> > </hostdev> > # virsh attach-device rhel hostdev.xml > error: Failed to attach device from hostdev.xml > error: internal error: unable to execute QEMU command 'device_add': vfio > error: 0000:07:00.0: group 2 is not viable > > > ------- > Best Regards, > Yalan Zhang > IRC: yalzhang > Internal phone: 8389413 > > > _______________________________________________ > libvirt-users mailing list > libvirt-users@redhat.com > https://www.redhat.com/mailman/listinfo/libvirt-users >