libvirt users - Apr 2020 - Not able to add pcie card to guest: Operation not permitted

Let's say I have libvirt

[root@vmhost2 ~]# virsh version
Compiled against library: libvirt 4.5.0
Using library: libvirt 4.5.0
Using API: QEMU 4.5.0
Running hypervisor: QEMU 2.12.0
[root@vmhost2 ~]#

running on centos 8 and then I have this card

[root@vmhost2 ~]# virsh nodedev-dumpxml pci_0000_01_00_0
<device>
  <name>pci_0000_01_00_0</name>
  <path>/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0</path>
  <parent>pci_0000_00_01_0</parent>
  <driver>
    <name>mlx4_core</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>1</bus>
    <slot>0</slot>
    <function>0</function>
    <product id='0x1003'>MT27500 Family
[ConnectX-3]</product>
    <vendor id='0x15b3'>Mellanox Technologies</vendor>
    <iommuGroup number='1'>
      <address domain='0x0000' bus='0x00' slot='0x01'
function='0x0'/>
      <address domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
    </iommuGroup>
    <pci-express>
      <link validity='cap' port='8' speed='8'
width='8'/>
      <link validity='sta' speed='5' width='8'/>
    </pci-express>
  </capability>
</device>

which I added to the guest (arch='x86_64'
machine='pc-q35-rhel7.6.0') as

    <hostdev mode='subsystem' type='pci'
managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01'
slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x08'
slot='0x00'
function='0x0'/>
    </hostdev>

When I try to start the guest I get the following error message:

[root@vmhost2 ~]# virsh start testfedora
error: Failed to start domain testfedora
error: internal error: qemu unexpectedly closed the monitor:
2020-04-24T20:01:35.341020Z qemu-kvm: -device
vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error:
0000:01:00.0: failed to setup INTx fd: Operation not permitted

[root@vmhost2 ~]#

Why is it telling me that is not permitted?

On Fri, 24 Apr 2020 at 21:10, Mauricio Tavares <raubvogel@gmail.com>
wrote:
> Let's say I have libvirt
>
> [root@vmhost2 ~]# virsh version
> [...]
Running hypervisor: QEMU 2.12.0
> [root@vmhost2 ~]#
> [...]
>
When I try to start the guest I get the following error message:
>
> [root@vmhost2 ~]# virsh start testfedora
> error: Failed to start domain testfedora
> error: internal error: qemu unexpectedly closed the monitor:
> 2020-04-24T20:01:35.341020Z qemu-kvm: -device
> vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error:
> 0000:01:00.0: failed to setup INTx fd: Operation not permitted
>
> [root@vmhost2 ~]#
>
> Why is it telling me that is not permitted?
>
> The guest will be running as qemu on the host.  Does qemu have appropriate
permissions in the host, and does that include in any hardening like
SElinux that you're running?

Cheers,

- Peter

On Fri, Apr 24, 2020 at 4:35 PM Peter Crowther
<peter.crowther@melandra.com> wrote:
>
> On Fri, 24 Apr 2020 at 21:10, Mauricio Tavares <raubvogel@gmail.com>
wrote:
>>
>> Let's say I have libvirt
>>
>> [root@vmhost2 ~]# virsh version
>> [...]
>>
>> Running hypervisor: QEMU 2.12.0
>> [root@vmhost2 ~]#
>> [...]
>
> When I try to start the guest I get the following error message:
>>
>>
>> [root@vmhost2 ~]# virsh start testfedora
>> error: Failed to start domain testfedora
>> error: internal error: qemu unexpectedly closed the monitor:
>> 2020-04-24T20:01:35.341020Z qemu-kvm: -device
>> vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error:
>> 0000:01:00.0: failed to setup INTx fd: Operation not permitted
>>
>> [root@vmhost2 ~]#
>>
>> Why is it telling me that is not permitted?
>>
> The guest will be running as qemu on the host.  Does qemu have appropriate
permissions in the host, and does that include in any hardening like SElinux
that you're running?
>
I tried with selinux in permissive mode to see if it made a
difference. Not much.

[root@vmhost2 ~]# getenforce
Permissive
[root@vmhost2 ~]# virsh start testfedora
error: Failed to start domain testfedora
error: internal error: qemu unexpectedly closed the monitor:
2020-04-25T00:43:36.621246Z qemu-kvm: -device
vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error:
0000:01:00.0: failed to setup INTx fd: Operation not permitted

[root@vmhost2 ~]#

For the fun of it, I swapped that card with another one (same speed,
number of ports, diff brand), so it is on th every sam epci slot:

[root@vmhost2 ~]# virsh nodedev-dumpxml pci_0000_01_00_0
<device>
  <name>pci_0000_01_00_0</name>
  <path>/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0</path>
  <parent>pci_0000_00_01_0</parent>
  <driver>
    <name>vfio-pci</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>1</bus>
    <slot>0</slot>
    <function>0</function>
    <product id='0x4000' />
    <vendor id='0x19ee'>Netronome Systems, Inc.</vendor>
    <capability type='virt_functions' maxCount='64'/>
    <iommuGroup number='1'>
      <address domain='0x0000' bus='0x00' slot='0x01'
function='0x0'/>
      <address domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
    </iommuGroup>
    <pci-express>
      <link validity='cap' port='0' speed='8'
width='8'/>
      <link validity='sta' speed='2.5' width='8'/>
    </pci-express>
  </capability>
</device>


[root@vmhost2 ~]#

And it starts without an issue:

[root@vmhost2 ~]# virsh start testfedora
Domain testfedora started

[root@vmhost2 ~]#

Inside the guest:

[root@testfedora ~]# dmesg |grep -i netronome
[   12.327316] nfp: NFP PCIe Driver, Copyright (C) 2014-2017 Netronome Systems
[   12.335036] nfp 0000:07:00.0: Netronome Flow Processor
NFP4000/NFP5000/NFP6000 PCIe Card Probe
[root@testfedora ~]#

so I do not know what is going on.
> Cheers,
>
> - Peter
>