libvirt users - Mar 2014 - PCI Passthrough of 2 identical devices

hi,

i have a small trouble with pci-passthrough.

i have a working configuration passing a tv card into the guest, all is fine and
dandy.
as soon as i add my second identical card into the host i cant start the guest
anymore, whether i add the second card to the guest or not. error message is
identical in both cases.

message of virt-manager|virsh|libvirtd.log:
> libvirtError: internal error: Unable to reset PCI device 0000:03:00.0:
internal error: Active 0000:03:01.0 devices on bus with 0000:03:00.0, not doing
bus reset
the host is a plain kvm server, no services or apps whatsoever accessing the two
devices.
the other guests (firewall and fileserver) do not use the cards either.

if i (soft) remove the 2nd card via
> echo -n 1 > /sys/bus/pci/devices/0000\:03\:01.0/remove
i can start the guest with the 1st card assigned.

thx for anyone looking into this.

wbr,tja..


PS:
host:
> root@father:~# lspci
> 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM
Controller (rev 06)
> 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th
Gen Core Processor Integrated Graphics Controller (rev 06)
> 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset
Family USB xHCI (rev 05)
> 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series
Chipset Family MEI Controller #1 (rev 04)
> 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V
(rev 05)
> 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset
Family USB EHCI #2 (rev 05)
> 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family
PCI Express Root Port #1 (rev d5)
> 00:1c.3 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d5)
> 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset
Family USB EHCI #1 (rev 05)
> 00:1f.0 ISA bridge: Intel Corporation H87 Express LPC Controller (rev 05)
> 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset
Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
> 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus
Controller (rev 05)
> 02:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge
(rev 03)
> 03:00.0 Multimedia controller: Philips Semiconductors SAA7146 (rev 01)
> 03:01.0 Multimedia controller: Philips Semiconductors SAA7146 (rev 01)
> root@father:~# cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=13.10
> DISTRIB_CODENAME=saucy
> DISTRIB_DESCRIPTION="Ubuntu 13.10"
> root@father:~# uname -a
> Linux father 3.11.0-18-generic #32-Ubuntu SMP Tue Feb 18 21:11:14 UTC 2014
x86_64 x86_64 x86_64 GNU/Linux
> root@father:~# libvirtd --version
> libvirtd (libvirt) 1.1.1
guest:
> root@father:~# cat /etc/libvirt/qemu/tvBackend.xml 
> <!--
> WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
> OVERWRITTEN AND LOST. Changes to this xml configuration should be made
using:
>   virsh edit tvBackend
> or other application using the libvirt API.
> -->
> 
> <domain type='kvm'>
>   <name>tvBackend</name>
>   <uuid>f647bb10-7f9a-f94c-33b9-3d99e8e753e0</uuid>
>   <memory unit='KiB'>1048576</memory>
>   <currentMemory unit='KiB'>1048576</currentMemory>
>   <vcpu placement='static'>2</vcpu>
>   <os>
>     <type arch='x86_64'
machine='pc-i440fx-1.5'>hvm</type>
>     <boot dev='hd'/>
>   </os>
>   <features>
>     <acpi/>
>     <apic/>
>     <pae/>
>   </features>
>   <clock offset='utc'/>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>restart</on_crash>
>   <devices>
>     <emulator>/usr/bin/kvm-spice</emulator>
>     <disk type='file' device='disk'>
>       <driver name='qemu' type='raw'/>
>       <source file='/kvm/tvBackend.img'/>
>       <target dev='vda' bus='virtio'/>
>       <address type='pci' domain='0x0000'
bus='0x00' slot='0x04' function='0x0'/>
>     </disk>
>     <disk type='block' device='cdrom'>
>       <driver name='qemu' type='raw'/>
>       <target dev='hdc' bus='ide'/>
>       <readonly/>
>       <address type='drive' controller='0'
bus='1' target='0' unit='0'/>
>     </disk>
>     <controller type='usb' index='0'>
>       <address type='pci' domain='0x0000'
bus='0x00' slot='0x01' function='0x2'/>
>     </controller>
>     <controller type='pci' index='0'
model='pci-root'/>
>     <controller type='ide' index='0'>
>       <address type='pci' domain='0x0000'
bus='0x00' slot='0x01' function='0x1'/>
>     </controller>
>     <interface type='bridge'>
>       <mac address='52:54:00:10:26:ec'/>
>       <source bridge='brlan'/>
>       <model type='virtio'/>
>       <address type='pci' domain='0x0000'
bus='0x00' slot='0x03' function='0x0'/>
>     </interface>
>     <serial type='pty'>
>       <target port='0'/>
>     </serial>
>     <console type='pty'>
>       <target type='serial' port='0'/>
>     </console>
>     <input type='mouse' bus='ps2'/>
>     <graphics type='vnc' port='-1'
autoport='yes'/>
>     <video>
>       <model type='vga' vram='9216'
heads='1'/>
>       <address type='pci' domain='0x0000'
bus='0x00' slot='0x02' function='0x0'/>
>     </video>
>     <hostdev mode='subsystem' type='pci'
managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x03'
slot='0x00' function='0x0'/>
>       </source>
>       <address type='pci' domain='0x0000'
bus='0x00' slot='0x06' function='0x0'/>
>     </hostdev>
>     <memballoon model='virtio'>
>       <address type='pci' domain='0x0000'
bus='0x00' slot='0x05' function='0x0'/>
>     </memballoon>
>   </devices>
> </domain>
-- 
thomas jagoditsch - tjaSoft
softWareEntwicklung - netzWerkManagement

On 03/12/2014 06:29 AM, Thomas Jagoditsch wrote:
> hi,
>
> i have a small trouble with pci-passthrough.
>
> i have a working configuration passing a tv card into the guest, all is
fine and dandy.
> as soon as i add my second identical card into the host i cant start the
guest anymore, whether i add the second card to the guest or not. error message
is identical in both cases.
>
> message of virt-manager|virsh|libvirtd.log:
>> libvirtError: internal error: Unable to reset PCI device 0000:03:00.0:
internal error: Active 0000:03:01.0 devices on bus with 0000:03:00.0, not doing
bus reset
> the host is a plain kvm server, no services or apps whatsoever accessing
the two devices.
> the other guests (firewall and fileserver) do not use the cards either.
The problem is that for kvm device assignment to work properly, the
device needs to be reset by libvirt after detaching it from the host
driver and before passing it to kvm, and these devices you are trying to
reset support neither "function level reset" nor "power
management
reset", so libvirt must fallback to resetting the entire bus where the
device is plugged in. But of course it can't do that if the bus contains
other devices that are in use by the host or by other guests.

If your host OS is new enough to support vfio device assignment, I would
suggest using that instead (if you update to the latest upstrem libvirt
it will automatically use VFIO if it is available, otherwise you can add
"<driver name='vfio'/>" to the device definition to
assure that it will
either use vfio or fail. You will also probably need to run "modprobe
vfio" before starting the guest). The reason I suggest this is that VFIO
will automatically handle resetting the assigned devices whenever
necessary, compared to the old KVM device assignment, where libvirt must
always reset the device because it has no better information about
whether or not it is really necessary.

Another thing you can try is plugging one of the cards into a different
slot - if you can find a slot that is on a different bus, then libvirt
will be able to reset the bus containing the card that will be assigned
(since there won't be any other active devices on that same bus).

>
> if i (soft) remove the 2nd card via
>> echo -n 1 > /sys/bus/pci/devices/0000\:03\:01.0/remove
> i can start the guest with the 1st card assigned.
>
> thx for anyone looking into this.
libvirt attempts to reset the
>
> wbr,tja..
>
>
> PS:
> host:
>> root@father:~# lspci
>> 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM
Controller (rev 06)
>> 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200
v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
>> 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset
Family USB xHCI (rev 05)
>> 00:16.0 Communication controller: Intel Corporation 8 Series/C220
Series Chipset Family MEI Controller #1 (rev 04)
>> 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection
I217-V (rev 05)
>> 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset
Family USB EHCI #2 (rev 05)
>> 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
Family PCI Express Root Port #1 (rev d5)
>> 00:1c.3 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d5)
>> 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset
Family USB EHCI #1 (rev 05)
>> 00:1f.0 ISA bridge: Intel Corporation H87 Express LPC Controller (rev
05)
>> 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset
Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
>> 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family
SMBus Controller (rev 05)
>> 02:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI
Bridge (rev 03)
>> 03:00.0 Multimedia controller: Philips Semiconductors SAA7146 (rev 01)
>> 03:01.0 Multimedia controller: Philips Semiconductors SAA7146 (rev 01)
>> root@father:~# cat /etc/lsb-release
>> DISTRIB_ID=Ubuntu
>> DISTRIB_RELEASE=13.10
>> DISTRIB_CODENAME=saucy
>> DISTRIB_DESCRIPTION="Ubuntu 13.10"
>> root@father:~# uname -a
>> Linux father 3.11.0-18-generic #32-Ubuntu SMP Tue Feb 18 21:11:14 UTC
2014 x86_64 x86_64 x86_64 GNU/Linux
>> root@father:~# libvirtd --version
>> libvirtd (libvirt) 1.1.1
> guest:
>> root@father:~# cat /etc/libvirt/qemu/tvBackend.xml 
>> <!--
>> WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
>> OVERWRITTEN AND LOST. Changes to this xml configuration should be made
using:
>>   virsh edit tvBackend
>> or other application using the libvirt API.
>> -->
>>
>> <domain type='kvm'>
>>   <name>tvBackend</name>
>>   <uuid>f647bb10-7f9a-f94c-33b9-3d99e8e753e0</uuid>
>>   <memory unit='KiB'>1048576</memory>
>>   <currentMemory unit='KiB'>1048576</currentMemory>
>>   <vcpu placement='static'>2</vcpu>
>>   <os>
>>     <type arch='x86_64'
machine='pc-i440fx-1.5'>hvm</type>
>>     <boot dev='hd'/>
>>   </os>
>>   <features>
>>     <acpi/>
>>     <apic/>
>>     <pae/>
>>   </features>
>>   <clock offset='utc'/>
>>   <on_poweroff>destroy</on_poweroff>
>>   <on_reboot>restart</on_reboot>
>>   <on_crash>restart</on_crash>
>>   <devices>
>>     <emulator>/usr/bin/kvm-spice</emulator>
>>     <disk type='file' device='disk'>
>>       <driver name='qemu' type='raw'/>
>>       <source file='/kvm/tvBackend.img'/>
>>       <target dev='vda' bus='virtio'/>
>>       <address type='pci' domain='0x0000'
bus='0x00' slot='0x04' function='0x0'/>
>>     </disk>
>>     <disk type='block' device='cdrom'>
>>       <driver name='qemu' type='raw'/>
>>       <target dev='hdc' bus='ide'/>
>>       <readonly/>
>>       <address type='drive' controller='0'
bus='1' target='0' unit='0'/>
>>     </disk>
>>     <controller type='usb' index='0'>
>>       <address type='pci' domain='0x0000'
bus='0x00' slot='0x01' function='0x2'/>
>>     </controller>
>>     <controller type='pci' index='0'
model='pci-root'/>
>>     <controller type='ide' index='0'>
>>       <address type='pci' domain='0x0000'
bus='0x00' slot='0x01' function='0x1'/>
>>     </controller>
>>     <interface type='bridge'>
>>       <mac address='52:54:00:10:26:ec'/>
>>       <source bridge='brlan'/>
>>       <model type='virtio'/>
>>       <address type='pci' domain='0x0000'
bus='0x00' slot='0x03' function='0x0'/>
>>     </interface>
>>     <serial type='pty'>
>>       <target port='0'/>
>>     </serial>
>>     <console type='pty'>
>>       <target type='serial' port='0'/>
>>     </console>
>>     <input type='mouse' bus='ps2'/>
>>     <graphics type='vnc' port='-1'
autoport='yes'/>
>>     <video>
>>       <model type='vga' vram='9216'
heads='1'/>
>>       <address type='pci' domain='0x0000'
bus='0x00' slot='0x02' function='0x0'/>
>>     </video>
>>     <hostdev mode='subsystem' type='pci'
managed='yes'>
>>       <source>
>>         <address domain='0x0000' bus='0x03'
slot='0x00' function='0x0'/>
>>       </source>
>>       <address type='pci' domain='0x0000'
bus='0x00' slot='0x06' function='0x0'/>
>>     </hostdev>
>>     <memballoon model='virtio'>
>>       <address type='pci' domain='0x0000'
bus='0x00' slot='0x05' function='0x0'/>
>>     </memballoon>
>>   </devices>
>> </domain>