Mauricio Tavares
2020-Apr-24 20:09 UTC
Not able to add pcie card to guest: Operation not permitted
Let's say I have libvirt
[root@vmhost2 ~]# virsh version
Compiled against library: libvirt 4.5.0
Using library: libvirt 4.5.0
Using API: QEMU 4.5.0
Running hypervisor: QEMU 2.12.0
[root@vmhost2 ~]#
running on centos 8 and then I have this card
[root@vmhost2 ~]# virsh nodedev-dumpxml pci_0000_01_00_0
<device>
<name>pci_0000_01_00_0</name>
<path>/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0</path>
<parent>pci_0000_00_01_0</parent>
<driver>
<name>mlx4_core</name>
</driver>
<capability type='pci'>
<domain>0</domain>
<bus>1</bus>
<slot>0</slot>
<function>0</function>
<product id='0x1003'>MT27500 Family
[ConnectX-3]</product>
<vendor id='0x15b3'>Mellanox Technologies</vendor>
<iommuGroup number='1'>
<address domain='0x0000' bus='0x00' slot='0x01'
function='0x0'/>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
</iommuGroup>
<pci-express>
<link validity='cap' port='8' speed='8'
width='8'/>
<link validity='sta' speed='5' width='8'/>
</pci-express>
</capability>
</device>
which I added to the guest (arch='x86_64'
machine='pc-q35-rhel7.6.0') as
<hostdev mode='subsystem' type='pci'
managed='yes'>
<source>
<address domain='0x0000' bus='0x01'
slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x08'
slot='0x00'
function='0x0'/>
</hostdev>
When I try to start the guest I get the following error message:
[root@vmhost2 ~]# virsh start testfedora
error: Failed to start domain testfedora
error: internal error: qemu unexpectedly closed the monitor:
2020-04-24T20:01:35.341020Z qemu-kvm: -device
vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error:
0000:01:00.0: failed to setup INTx fd: Operation not permitted
[root@vmhost2 ~]#
Why is it telling me that is not permitted?
Peter Crowther
2020-Apr-24 20:34 UTC
Re: Not able to add pcie card to guest: Operation not permitted
On Fri, 24 Apr 2020 at 21:10, Mauricio Tavares <raubvogel@gmail.com> wrote:> Let's say I have libvirt > > [root@vmhost2 ~]# virsh version > [...]Running hypervisor: QEMU 2.12.0> [root@vmhost2 ~]# > [...] >When I try to start the guest I get the following error message:> > [root@vmhost2 ~]# virsh start testfedora > error: Failed to start domain testfedora > error: internal error: qemu unexpectedly closed the monitor: > 2020-04-24T20:01:35.341020Z qemu-kvm: -device > vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error: > 0000:01:00.0: failed to setup INTx fd: Operation not permitted > > [root@vmhost2 ~]# > > Why is it telling me that is not permitted? > > The guest will be running as qemu on the host. Does qemu have appropriatepermissions in the host, and does that include in any hardening like SElinux that you're running? Cheers, - Peter
Mauricio Tavares
2020-Apr-25 01:30 UTC
Re: Not able to add pcie card to guest: Operation not permitted
On Fri, Apr 24, 2020 at 4:35 PM Peter Crowther <peter.crowther@melandra.com> wrote:> > On Fri, 24 Apr 2020 at 21:10, Mauricio Tavares <raubvogel@gmail.com> wrote: >> >> Let's say I have libvirt >> >> [root@vmhost2 ~]# virsh version >> [...] >> >> Running hypervisor: QEMU 2.12.0 >> [root@vmhost2 ~]# >> [...] > > When I try to start the guest I get the following error message: >> >> >> [root@vmhost2 ~]# virsh start testfedora >> error: Failed to start domain testfedora >> error: internal error: qemu unexpectedly closed the monitor: >> 2020-04-24T20:01:35.341020Z qemu-kvm: -device >> vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error: >> 0000:01:00.0: failed to setup INTx fd: Operation not permitted >> >> [root@vmhost2 ~]# >> >> Why is it telling me that is not permitted? >> > The guest will be running as qemu on the host. Does qemu have appropriate permissions in the host, and does that include in any hardening like SElinux that you're running? >I tried with selinux in permissive mode to see if it made a difference. Not much. [root@vmhost2 ~]# getenforce Permissive [root@vmhost2 ~]# virsh start testfedora error: Failed to start domain testfedora error: internal error: qemu unexpectedly closed the monitor: 2020-04-25T00:43:36.621246Z qemu-kvm: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error: 0000:01:00.0: failed to setup INTx fd: Operation not permitted [root@vmhost2 ~]# For the fun of it, I swapped that card with another one (same speed, number of ports, diff brand), so it is on th every sam epci slot: [root@vmhost2 ~]# virsh nodedev-dumpxml pci_0000_01_00_0 <device> <name>pci_0000_01_00_0</name> <path>/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0</path> <parent>pci_0000_00_01_0</parent> <driver> <name>vfio-pci</name> </driver> <capability type='pci'> <domain>0</domain> <bus>1</bus> <slot>0</slot> <function>0</function> <product id='0x4000' /> <vendor id='0x19ee'>Netronome Systems, Inc.</vendor> <capability type='virt_functions' maxCount='64'/> <iommuGroup number='1'> <address domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </iommuGroup> <pci-express> <link validity='cap' port='0' speed='8' width='8'/> <link validity='sta' speed='2.5' width='8'/> </pci-express> </capability> </device> [root@vmhost2 ~]# And it starts without an issue: [root@vmhost2 ~]# virsh start testfedora Domain testfedora started [root@vmhost2 ~]# Inside the guest: [root@testfedora ~]# dmesg |grep -i netronome [ 12.327316] nfp: NFP PCIe Driver, Copyright (C) 2014-2017 Netronome Systems [ 12.335036] nfp 0000:07:00.0: Netronome Flow Processor NFP4000/NFP5000/NFP6000 PCIe Card Probe [root@testfedora ~]# so I do not know what is going on.> Cheers, > > - Peter >