Ivan Kabaivanov
2014-Jan-25 16:54 UTC
[libvirt-users] intel quad gigabit nic and pci passthrough
Hi all
I have a very weird case of pci passthrough.
I have a machine with 7 network interfaces, all of them intel. Four of them are
on one quad giga ethernet device. If I manually unbind the devices and allow
qemu to use them, with intel IOMMU working, everything works like a charm.
Here's how I do it manually:
root@kybrat (x86_64) ~]$ lspci -nn | grep net
00:19.0 Ethernet controller [0200]: Intel Corporation 82579V Gigabit Network
Connection [8086:1503] (rev 06)
02:00.0 Ethernet controller [0200]: Intel Corporation 82572EI Gigabit Ethernet
Controller (Copper) [8086:107d] (rev 06)
05:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet
Controller [8086:10a4] (rev 06)
05:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet
Controller [8086:10a4] (rev 06)
06:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet
Controller [8086:10a4] (rev 06)
06:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet
Controller [8086:10a4] (rev 06)
0c:00.0 Ethernet controller [0200]: Intel Corporation 82572EI Gigabit Ethernet
Controller (Copper) [8086:107d] (rev 06)
echo "8086 10a4" > /sys/bus/pci/drivers/pci-stub/new_id
echo "8086 107d" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:05:00.0" >
/sys/bus/pci/devices/0000\:05\:00.0/driver/unbind
echo "0000:05:00.0" > /sys/bus/pci/drivers/pci-stub/bind
echo "0000:05:00.1" >
/sys/bus/pci/devices/0000\:05\:00.1/driver/unbind
echo "0000:05:00.1" > /sys/bus/pci/drivers/pci-stub/bind
echo "0000:06:00.0" >
/sys/bus/pci/devices/0000\:06\:00.0/driver/unbind
echo "0000:06:00.0" > /sys/bus/pci/drivers/pci-stub/bind
echo "0000:06:00.1" >
/sys/bus/pci/devices/0000\:06\:00.1/driver/unbind
echo "0000:06:00.1" > /sys/bus/pci/drivers/pci-stub/bind
echo "0000:0c:00.0" >
/sys/bus/pci/devices/0000\:0c\:00.0/driver/unbind
echo "0000:0c:00.0" > /sys/bus/pci/drivers/pci-stub/bind
qemu-system-i386 -enable-kvm \
-m 2048 \
-cpu host \
-machine pc-q35-1.6 \
-drive file=/media/virtual/krym.qcow2,if=virtio \
-net none \
-device pci-assign,host=06:00.1,rombar=0 \
-device pci-assign,host=05:00.0,rombar=0 \
-device pci-assign,host=06:00.0,rombar=0 \
-device pci-assign,host=05:00.1,rombar=0 \
-device pci-assign,host=0c:00.0,rombar=0 \
-cdrom /media/virtual/asg-9.107-33.1.iso \
-boot c \
-vnc :0 \
-runas virtual \
-monitor unix:/media/virtual/control/krym/monitor,server,nowait
\
-rtc base=localtime,clock=host,driftfix=none
This starts qemu and pci passthrough works nicely.
Now, if I try to do the same thing through libvirt, virt-install doesn't
like the quad ethernet device pci passthrough and I get the error below:
virt-install --name=krym --ram=2048 --vcpus=1 --cpu=host --import
--os-type=linux --disk path=/media/virtual/krym.qcow2,format=qcow2,bus=virtio
--nonetworks --graphics vnc,port=5900 --virt-type=kvm --machine=q35 --accelerate
--host-device=pci_0000_06_00_1 --host-device=pci_0000_05_00_0
--host-device=pci_0000_06_00_0 --host-device=pci_0000_05_00_1
--host-device=pci_0000_0c_00_0
Starting install...
ERROR Requested operation is not valid: PCI device 0000:06:00.1 is not
assignable
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
virsh --connect qemu:///system start krym
otherwise, please restart your installation.
If I try to use just one NIC (any NIC, as long as it's not the quad NIC),
things do work:
virt-install --name=krym --ram=2048 --vcpus=1 --cpu=host --import
--os-type=linux --disk path=/media/virtual/krym.qcow2,format=qcow2,bus=virtio
--nonetworks --graphics vnc,port=5900 --virt-type=kvm --machine=q35 --accelerate
--host-device=pci_0000_0c_00_0
Starting install...
Creating domain...
| 0 B 00:00:01
Cannot open display:
Run 'virt-viewer --help' to see a full list of available command line
options
Domain creation completed. You can restart your domain by running:
virsh --connect qemu:///system start krym
So I guess libvirt has some problem with the intel quad NIC.
I will appreciate any help.
Thanks,
IvanK.
Laine Stump
2014-Jan-28 10:58 UTC
Re: [libvirt-users] intel quad gigabit nic and pci passthrough
On 01/25/2014 06:54 PM, Ivan Kabaivanov wrote:> Hi all > > I have a very weird case of pci passthrough. > > I have a machine with 7 network interfaces, all of them intel. Four of them are on one quad giga ethernet device. If I manually unbind the devices and allow qemu to use them, with intel IOMMU working, everything works like a charm. Here's how I do it manually: > > root@kybrat (x86_64) ~]$ lspci -nn | grep net > 00:19.0 Ethernet controller [0200]: Intel Corporation 82579V Gigabit Network Connection [8086:1503] (rev 06) > 02:00.0 Ethernet controller [0200]: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) [8086:107d] (rev 06) > 05:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:10a4] (rev 06) > 05:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:10a4] (rev 06) > 06:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:10a4] (rev 06) > 06:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:10a4] (rev 06) > 0c:00.0 Ethernet controller [0200]: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) [8086:107d] (rev 06) > > > echo "8086 10a4" > /sys/bus/pci/drivers/pci-stub/new_id > echo "8086 107d" > /sys/bus/pci/drivers/pci-stub/new_id > echo "0000:05:00.0" > /sys/bus/pci/devices/0000\:05\:00.0/driver/unbind > echo "0000:05:00.0" > /sys/bus/pci/drivers/pci-stub/bind > echo "0000:05:00.1" > /sys/bus/pci/devices/0000\:05\:00.1/driver/unbind > echo "0000:05:00.1" > /sys/bus/pci/drivers/pci-stub/bind > echo "0000:06:00.0" > /sys/bus/pci/devices/0000\:06\:00.0/driver/unbind > echo "0000:06:00.0" > /sys/bus/pci/drivers/pci-stub/bind > echo "0000:06:00.1" > /sys/bus/pci/devices/0000\:06\:00.1/driver/unbind > echo "0000:06:00.1" > /sys/bus/pci/drivers/pci-stub/bind > echo "0000:0c:00.0" > /sys/bus/pci/devices/0000\:0c\:00.0/driver/unbind > echo "0000:0c:00.0" > /sys/bus/pci/drivers/pci-stub/bind > > qemu-system-i386 -enable-kvm \ > -m 2048 \ > -cpu host \ > -machine pc-q35-1.6 \ > -drive file=/media/virtual/krym.qcow2,if=virtio \ > -net none \ > -device pci-assign,host=06:00.1,rombar=0 \ > -device pci-assign,host=05:00.0,rombar=0 \ > -device pci-assign,host=06:00.0,rombar=0 \ > -device pci-assign,host=05:00.1,rombar=0 \ > -device pci-assign,host=0c:00.0,rombar=0 \ > -cdrom /media/virtual/asg-9.107-33.1.iso \ > -boot c \ > -vnc :0 \ > -runas virtual \ > -monitor unix:/media/virtual/control/krym/monitor,server,nowait \ > -rtc base=localtime,clock=host,driftfix=none > > This starts qemu and pci passthrough works nicely. > > Now, if I try to do the same thing through libvirt, virt-install doesn't like the quad ethernet device pci passthrough and I get the error below: > > virt-install --name=krym --ram=2048 --vcpus=1 --cpu=host --import --os-type=linux --disk path=/media/virtual/krym.qcow2,format=qcow2,bus=virtio --nonetworks --graphics vnc,port=5900 --virt-type=kvm --machine=q35 --accelerate --host-device=pci_0000_06_00_1 --host-device=pci_0000_05_00_0 --host-device=pci_0000_06_00_0 --host-device=pci_0000_05_00_1 --host-device=pci_0000_0c_00_0 > > Starting install... > ERROR Requested operation is not valid: PCI device 0000:06:00.1 is not assignable > Domain installation does not appear to have been successful. > If it was, you can restart your domain by running: > virsh --connect qemu:///system start krym > otherwise, please restart your installation.More information is needed to make a guess at the source of the problem: 1) What distro/version of Linux are you running. 2) What version of libvirt? 3) After the failed virt-install command, if the new domain is still defined, can you provide the output of: virsh dumpxml $domain (if virt-install is automatically undefining the failed domain, then you can get the xml for the defined domain by appending "--print-xml" to the virt-install commandline) 4) Also, send the last several lines of /var/log/libvirt/qemu/$domain.log, in particular the qemu commandline that was executed to start the domain, and any subsequent error messages. (Just one possibility - since you are successfully using the q35 machinetype, it's highly likely that your OS is new enough that it supports vfio for device passthrough; vfio is more restrictive about what devices can be assigned to a guest, sometimes requiring seemingly unrelated devices to be assigned to the same guest, and the most recent libvirts will prefer vfio over legacy kvm device assignment when available. So it's possible that your manual running of qemu-kvm works because you are forcing use of legacy kvm assignment, and when libvirt attempts to use vfio it is failing. This will be made apparent by the information I requested above.> > > If I try to use just one NIC (any NIC, as long as it's not the quad NIC), things do work: > > virt-install --name=krym --ram=2048 --vcpus=1 --cpu=host --import --os-type=linux --disk path=/media/virtual/krym.qcow2,format=qcow2,bus=virtio --nonetworks --graphics vnc,port=5900 --virt-type=kvm --machine=q35 --accelerate --host-device=pci_0000_0c_00_0 > > Starting install... > Creating domain... | 0 B 00:00:01 > Cannot open display: > Run 'virt-viewer --help' to see a full list of available command line options > Domain creation completed. You can restart your domain by running: > virsh --connect qemu:///system start krym > > > > So I guess libvirt has some problem with the intel quad NIC. > > I will appreciate any help. > > Thanks, > IvanK. > > _______________________________________________ > libvirt-users mailing list > libvirt-users@redhat.com > https://www.redhat.com/mailman/listinfo/libvirt-users > >
Ivan Kabaivanov
2014-Feb-02 11:16 UTC
Re: [libvirt-users] intel quad gigabit nic and pci passthrough
Laine, thanks for your reply. Following your suggestion to check the last few lines of the domain logs, I found this snippet: 2014-01-14 00:08:52.281+0000: 27653: error : virPCIDeviceIsAssignable:2223 : internal error: Device 0000:06:00.1 is behind a switch lacking ACS and cannot be assigned Google told me a very simple solution: enable relaxed_acs_check = 1 in qemu.conf. Now it works as expected. Thanks, IvanK. ----- Original Message ----- From: "Laine Stump" <laine@laine.org> To: libvirt-users@redhat.com Cc: chepati@yahoo.com Sent: Tuesday, January 28, 2014 12:58:05 PM Subject: Re: [libvirt-users] intel quad gigabit nic and pci passthrough On 01/25/2014 06:54 PM, Ivan Kabaivanov wrote:> Hi all > > I have a very weird case of pci passthrough. > > I have a machine with 7 network interfaces, all of them intel. Four of them are on one quad giga ethernet device. If I manually unbind the devices and allow qemu to use them, with intel IOMMU working, everything works like a charm. Here's how I do it manually: > > root@kybrat (x86_64) ~]$ lspci -nn | grep net > 00:19.0 Ethernet controller [0200]: Intel Corporation 82579V Gigabit Network Connection [8086:1503] (rev 06) > 02:00.0 Ethernet controller [0200]: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) [8086:107d] (rev 06) > 05:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:10a4] (rev 06) > 05:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:10a4] (rev 06) > 06:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:10a4] (rev 06) > 06:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:10a4] (rev 06) > 0c:00.0 Ethernet controller [0200]: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) [8086:107d] (rev 06) > > > echo "8086 10a4" > /sys/bus/pci/drivers/pci-stub/new_id > echo "8086 107d" > /sys/bus/pci/drivers/pci-stub/new_id > echo "0000:05:00.0" > /sys/bus/pci/devices/0000\:05\:00.0/driver/unbind > echo "0000:05:00.0" > /sys/bus/pci/drivers/pci-stub/bind > echo "0000:05:00.1" > /sys/bus/pci/devices/0000\:05\:00.1/driver/unbind > echo "0000:05:00.1" > /sys/bus/pci/drivers/pci-stub/bind > echo "0000:06:00.0" > /sys/bus/pci/devices/0000\:06\:00.0/driver/unbind > echo "0000:06:00.0" > /sys/bus/pci/drivers/pci-stub/bind > echo "0000:06:00.1" > /sys/bus/pci/devices/0000\:06\:00.1/driver/unbind > echo "0000:06:00.1" > /sys/bus/pci/drivers/pci-stub/bind > echo "0000:0c:00.0" > /sys/bus/pci/devices/0000\:0c\:00.0/driver/unbind > echo "0000:0c:00.0" > /sys/bus/pci/drivers/pci-stub/bind > > qemu-system-i386 -enable-kvm \ > -m 2048 \ > -cpu host \ > -machine pc-q35-1.6 \ > -drive file=/media/virtual/krym.qcow2,if=virtio \ > -net none \ > -device pci-assign,host=06:00.1,rombar=0 \ > -device pci-assign,host=05:00.0,rombar=0 \ > -device pci-assign,host=06:00.0,rombar=0 \ > -device pci-assign,host=05:00.1,rombar=0 \ > -device pci-assign,host=0c:00.0,rombar=0 \ > -cdrom /media/virtual/asg-9.107-33.1.iso \ > -boot c \ > -vnc :0 \ > -runas virtual \ > -monitor unix:/media/virtual/control/krym/monitor,server,nowait \ > -rtc base=localtime,clock=host,driftfix=none > > This starts qemu and pci passthrough works nicely. > > Now, if I try to do the same thing through libvirt, virt-install doesn't like the quad ethernet device pci passthrough and I get the error below: > > virt-install --name=krym --ram=2048 --vcpus=1 --cpu=host --import --os-type=linux --disk path=/media/virtual/krym.qcow2,format=qcow2,bus=virtio --nonetworks --graphics vnc,port=5900 --virt-type=kvm --machine=q35 --accelerate --host-device=pci_0000_06_00_1 --host-device=pci_0000_05_00_0 --host-device=pci_0000_06_00_0 --host-device=pci_0000_05_00_1 --host-device=pci_0000_0c_00_0 > > Starting install... > ERROR Requested operation is not valid: PCI device 0000:06:00.1 is not assignable > Domain installation does not appear to have been successful. > If it was, you can restart your domain by running: > virsh --connect qemu:///system start krym > otherwise, please restart your installation.More information is needed to make a guess at the source of the problem: 1) What distro/version of Linux are you running. 2) What version of libvirt? 3) After the failed virt-install command, if the new domain is still defined, can you provide the output of: virsh dumpxml $domain (if virt-install is automatically undefining the failed domain, then you can get the xml for the defined domain by appending "--print-xml" to the virt-install commandline) 4) Also, send the last several lines of /var/log/libvirt/qemu/$domain.log, in particular the qemu commandline that was executed to start the domain, and any subsequent error messages. (Just one possibility - since you are successfully using the q35 machinetype, it's highly likely that your OS is new enough that it supports vfio for device passthrough; vfio is more restrictive about what devices can be assigned to a guest, sometimes requiring seemingly unrelated devices to be assigned to the same guest, and the most recent libvirts will prefer vfio over legacy kvm device assignment when available. So it's possible that your manual running of qemu-kvm works because you are forcing use of legacy kvm assignment, and when libvirt attempts to use vfio it is failing. This will be made apparent by the information I requested above.> > > If I try to use just one NIC (any NIC, as long as it's not the quad NIC), things do work: > > virt-install --name=krym --ram=2048 --vcpus=1 --cpu=host --import --os-type=linux --disk path=/media/virtual/krym.qcow2,format=qcow2,bus=virtio --nonetworks --graphics vnc,port=5900 --virt-type=kvm --machine=q35 --accelerate --host-device=pci_0000_0c_00_0 > > Starting install... > Creating domain... | 0 B 00:00:01 > Cannot open display: > Run 'virt-viewer --help' to see a full list of available command line options > Domain creation completed. You can restart your domain by running: > virsh --connect qemu:///system start krym > > > > So I guess libvirt has some problem with the intel quad NIC. > > I will appreciate any help. > > Thanks, > IvanK. > > _______________________________________________ > libvirt-users mailing list > libvirt-users@redhat.com > https://www.redhat.com/mailman/listinfo/libvirt-users > >