thr3ads.net - libvirt users - Re: [libvirt-users] intel quad gigabit nic and pci passthrough [Jan 2014]

If this information is useful, please help other people find it:
Share via:

Ivan Kabaivanov

2014-Jan-25 16:54 UTC

[libvirt-users] intel quad gigabit nic and pci passthrough

Hi all

I have a very weird case of pci passthrough.

I have a machine with 7 network interfaces, all of them intel.  Four of them are
on one quad giga ethernet device.  If I manually unbind the devices and allow
qemu to use them, with intel IOMMU working, everything works like a charm. 
Here's how I do it manually:

root@kybrat (x86_64) ~]$ lspci -nn | grep net
00:19.0 Ethernet controller [0200]: Intel Corporation 82579V Gigabit Network
Connection [8086:1503] (rev 06)
02:00.0 Ethernet controller [0200]: Intel Corporation 82572EI Gigabit Ethernet
Controller (Copper) [8086:107d] (rev 06)
05:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet
Controller [8086:10a4] (rev 06)
05:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet
Controller [8086:10a4] (rev 06)
06:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet
Controller [8086:10a4] (rev 06)
06:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet
Controller [8086:10a4] (rev 06)
0c:00.0 Ethernet controller [0200]: Intel Corporation 82572EI Gigabit Ethernet
Controller (Copper) [8086:107d] (rev 06)


echo "8086 10a4" > /sys/bus/pci/drivers/pci-stub/new_id
echo "8086 107d" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:05:00.0" >
/sys/bus/pci/devices/0000\:05\:00.0/driver/unbind
echo "0000:05:00.0" > /sys/bus/pci/drivers/pci-stub/bind
echo "0000:05:00.1" >
/sys/bus/pci/devices/0000\:05\:00.1/driver/unbind
echo "0000:05:00.1" > /sys/bus/pci/drivers/pci-stub/bind
echo "0000:06:00.0" >
/sys/bus/pci/devices/0000\:06\:00.0/driver/unbind
echo "0000:06:00.0" > /sys/bus/pci/drivers/pci-stub/bind
echo "0000:06:00.1" >
/sys/bus/pci/devices/0000\:06\:00.1/driver/unbind
echo "0000:06:00.1" > /sys/bus/pci/drivers/pci-stub/bind
echo "0000:0c:00.0" >
/sys/bus/pci/devices/0000\:0c\:00.0/driver/unbind
echo "0000:0c:00.0" > /sys/bus/pci/drivers/pci-stub/bind

qemu-system-i386 -enable-kvm \
                 -m 2048 \
                 -cpu host \
                 -machine pc-q35-1.6 \
                 -drive file=/media/virtual/krym.qcow2,if=virtio \       
                 -net none \
                 -device pci-assign,host=06:00.1,rombar=0 \
                 -device pci-assign,host=05:00.0,rombar=0 \
                 -device pci-assign,host=06:00.0,rombar=0 \
                 -device pci-assign,host=05:00.1,rombar=0 \
                 -device pci-assign,host=0c:00.0,rombar=0 \
                 -cdrom /media/virtual/asg-9.107-33.1.iso \
                 -boot c \
                 -vnc :0 \                                             
                 -runas virtual \
                 -monitor unix:/media/virtual/control/krym/monitor,server,nowait
\
                 -rtc base=localtime,clock=host,driftfix=none

This starts qemu and pci passthrough works nicely.

Now, if I try to do the same thing through libvirt, virt-install doesn't
like the quad ethernet device pci passthrough and I get the error below:

virt-install --name=krym --ram=2048 --vcpus=1 --cpu=host --import
--os-type=linux --disk path=/media/virtual/krym.qcow2,format=qcow2,bus=virtio
--nonetworks --graphics vnc,port=5900 --virt-type=kvm --machine=q35 --accelerate
--host-device=pci_0000_06_00_1 --host-device=pci_0000_05_00_0
--host-device=pci_0000_06_00_0 --host-device=pci_0000_05_00_1
--host-device=pci_0000_0c_00_0

Starting install...
ERROR    Requested operation is not valid: PCI device 0000:06:00.1 is not
assignable
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu:///system start krym
otherwise, please restart your installation.


If I try to use just one NIC (any NIC, as long as it's not the quad NIC),
things do work:

virt-install --name=krym --ram=2048 --vcpus=1 --cpu=host --import
--os-type=linux --disk path=/media/virtual/krym.qcow2,format=qcow2,bus=virtio
--nonetworks --graphics vnc,port=5900 --virt-type=kvm --machine=q35 --accelerate
--host-device=pci_0000_0c_00_0

Starting install...
Creating domain...                                                              
|    0 B  00:00:01
Cannot open display: 
Run 'virt-viewer --help' to see a full list of available command line
options
Domain creation completed. You can restart your domain by running:
  virsh --connect qemu:///system start krym



So I guess libvirt has some problem with the intel quad NIC.

I will appreciate any help.

Thanks,
IvanK.

Laine Stump

2014-Jan-28 10:58 UTC

head link

Re: [libvirt-users] intel quad gigabit nic and pci passthrough

On 01/25/2014 06:54 PM, Ivan Kabaivanov wrote:> Hi all
>
> I have a very weird case of pci passthrough.
>
> I have a machine with 7 network interfaces, all of them intel.  Four of
them are on one quad giga ethernet device.  If I manually unbind the devices and
allow qemu to use them, with intel IOMMU working, everything works like a charm.
Here's how I do it manually:
>
> root@kybrat (x86_64) ~]$ lspci -nn | grep net
> 00:19.0 Ethernet controller [0200]: Intel Corporation 82579V Gigabit
Network Connection [8086:1503] (rev 06)
> 02:00.0 Ethernet controller [0200]: Intel Corporation 82572EI Gigabit
Ethernet Controller (Copper) [8086:107d] (rev 06)
> 05:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit
Ethernet Controller [8086:10a4] (rev 06)
> 05:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit
Ethernet Controller [8086:10a4] (rev 06)
> 06:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit
Ethernet Controller [8086:10a4] (rev 06)
> 06:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit
Ethernet Controller [8086:10a4] (rev 06)
> 0c:00.0 Ethernet controller [0200]: Intel Corporation 82572EI Gigabit
Ethernet Controller (Copper) [8086:107d] (rev 06)
>
>
> echo "8086 10a4" > /sys/bus/pci/drivers/pci-stub/new_id
> echo "8086 107d" > /sys/bus/pci/drivers/pci-stub/new_id
> echo "0000:05:00.0" >
/sys/bus/pci/devices/0000\:05\:00.0/driver/unbind
> echo "0000:05:00.0" > /sys/bus/pci/drivers/pci-stub/bind
> echo "0000:05:00.1" >
/sys/bus/pci/devices/0000\:05\:00.1/driver/unbind
> echo "0000:05:00.1" > /sys/bus/pci/drivers/pci-stub/bind
> echo "0000:06:00.0" >
/sys/bus/pci/devices/0000\:06\:00.0/driver/unbind
> echo "0000:06:00.0" > /sys/bus/pci/drivers/pci-stub/bind
> echo "0000:06:00.1" >
/sys/bus/pci/devices/0000\:06\:00.1/driver/unbind
> echo "0000:06:00.1" > /sys/bus/pci/drivers/pci-stub/bind
> echo "0000:0c:00.0" >
/sys/bus/pci/devices/0000\:0c\:00.0/driver/unbind
> echo "0000:0c:00.0" > /sys/bus/pci/drivers/pci-stub/bind
>
> qemu-system-i386 -enable-kvm \
>                  -m 2048 \
>                  -cpu host \
>                  -machine pc-q35-1.6 \
>                  -drive file=/media/virtual/krym.qcow2,if=virtio \       
>                  -net none \
>                  -device pci-assign,host=06:00.1,rombar=0 \
>                  -device pci-assign,host=05:00.0,rombar=0 \
>                  -device pci-assign,host=06:00.0,rombar=0 \
>                  -device pci-assign,host=05:00.1,rombar=0 \
>                  -device pci-assign,host=0c:00.0,rombar=0 \
>                  -cdrom /media/virtual/asg-9.107-33.1.iso \
>                  -boot c \
>                  -vnc :0 \                                             
>                  -runas virtual \
>                  -monitor
unix:/media/virtual/control/krym/monitor,server,nowait \
>                  -rtc base=localtime,clock=host,driftfix=none
>
> This starts qemu and pci passthrough works nicely.
>
> Now, if I try to do the same thing through libvirt, virt-install
doesn't like the quad ethernet device pci passthrough and I get the error
below:
>
> virt-install --name=krym --ram=2048 --vcpus=1 --cpu=host --import
--os-type=linux --disk path=/media/virtual/krym.qcow2,format=qcow2,bus=virtio
--nonetworks --graphics vnc,port=5900 --virt-type=kvm --machine=q35 --accelerate
--host-device=pci_0000_06_00_1 --host-device=pci_0000_05_00_0
--host-device=pci_0000_06_00_0 --host-device=pci_0000_05_00_1
--host-device=pci_0000_0c_00_0
>
> Starting install...
> ERROR    Requested operation is not valid: PCI device 0000:06:00.1 is not
assignable
> Domain installation does not appear to have been successful.
> If it was, you can restart your domain by running:
>   virsh --connect qemu:///system start krym
> otherwise, please restart your installation.
More information is needed to make a guess at the source of the problem:

1) What distro/version of Linux are you running.
2) What version of libvirt?
3) After the failed virt-install command, if the new domain is still
defined, can you provide the output of:

    virsh dumpxml $domain

(if virt-install is automatically undefining the failed domain, then you
can get the xml for the defined domain by appending "--print-xml" to
the
virt-install commandline)

4) Also, send the last several lines of
/var/log/libvirt/qemu/$domain.log, in particular the qemu commandline
that was executed to start the domain, and any subsequent error messages.

(Just one possibility - since you are successfully using the q35
machinetype, it's highly likely that your OS is new enough that it
supports vfio for device passthrough; vfio is more restrictive about
what devices can be assigned to a guest, sometimes requiring seemingly
unrelated devices to be assigned to the same guest, and the most recent
libvirts will prefer vfio over legacy kvm device assignment when
available. So it's possible that your manual running of qemu-kvm works
because you are forcing use of legacy kvm assignment, and when libvirt
attempts to use vfio it is failing. This will be made apparent by the
information I requested above.

>
>
> If I try to use just one NIC (any NIC, as long as it's not the quad
NIC), things do work:
>
> virt-install --name=krym --ram=2048 --vcpus=1 --cpu=host --import
--os-type=linux --disk path=/media/virtual/krym.qcow2,format=qcow2,bus=virtio
--nonetworks --graphics vnc,port=5900 --virt-type=kvm --machine=q35 --accelerate
--host-device=pci_0000_0c_00_0
>
> Starting install...
> Creating domain...                                                         
|    0 B  00:00:01
> Cannot open display: 
> Run 'virt-viewer --help' to see a full list of available command
line options
> Domain creation completed. You can restart your domain by running:
>   virsh --connect qemu:///system start krym
>
>
>
> So I guess libvirt has some problem with the intel quad NIC.
>
> I will appreciate any help.
>
> Thanks,
> IvanK.
>
> _______________________________________________
> libvirt-users mailing list
> libvirt-users@redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users
>
>

Ivan Kabaivanov

2014-Feb-02 11:16 UTC

head link

Re: [libvirt-users] intel quad gigabit nic and pci passthrough

Laine,

thanks for your reply.  Following your suggestion to check the last few lines of
the domain logs, I found this snippet:

2014-01-14 00:08:52.281+0000: 27653: error : virPCIDeviceIsAssignable:2223 :
internal error: Device 0000:06:00.1 is behind a switch lacking ACS and cannot be
assigned

Google told me a very simple solution: enable relaxed_acs_check = 1 in
qemu.conf.

Now it works as expected.

Thanks,
IvanK.




----- Original Message -----
From: "Laine Stump" <laine@laine.org>
To: libvirt-users@redhat.com
Cc: chepati@yahoo.com
Sent: Tuesday, January 28, 2014 12:58:05 PM
Subject: Re: [libvirt-users] intel quad gigabit nic and pci passthrough

On 01/25/2014 06:54 PM, Ivan Kabaivanov wrote:> Hi all
>
> I have a very weird case of pci passthrough.
>
> I have a machine with 7 network interfaces, all of them intel.  Four of
them are on one quad giga ethernet device.  If I manually unbind the devices and
allow qemu to use them, with intel IOMMU working, everything works like a charm.
Here's how I do it manually:
>
> root@kybrat (x86_64) ~]$ lspci -nn | grep net
> 00:19.0 Ethernet controller [0200]: Intel Corporation 82579V Gigabit
Network Connection [8086:1503] (rev 06)
> 02:00.0 Ethernet controller [0200]: Intel Corporation 82572EI Gigabit
Ethernet Controller (Copper) [8086:107d] (rev 06)
> 05:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit
Ethernet Controller [8086:10a4] (rev 06)
> 05:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit
Ethernet Controller [8086:10a4] (rev 06)
> 06:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit
Ethernet Controller [8086:10a4] (rev 06)
> 06:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit
Ethernet Controller [8086:10a4] (rev 06)
> 0c:00.0 Ethernet controller [0200]: Intel Corporation 82572EI Gigabit
Ethernet Controller (Copper) [8086:107d] (rev 06)
>
>
> echo "8086 10a4" > /sys/bus/pci/drivers/pci-stub/new_id
> echo "8086 107d" > /sys/bus/pci/drivers/pci-stub/new_id
> echo "0000:05:00.0" >
/sys/bus/pci/devices/0000\:05\:00.0/driver/unbind
> echo "0000:05:00.0" > /sys/bus/pci/drivers/pci-stub/bind
> echo "0000:05:00.1" >
/sys/bus/pci/devices/0000\:05\:00.1/driver/unbind
> echo "0000:05:00.1" > /sys/bus/pci/drivers/pci-stub/bind
> echo "0000:06:00.0" >
/sys/bus/pci/devices/0000\:06\:00.0/driver/unbind
> echo "0000:06:00.0" > /sys/bus/pci/drivers/pci-stub/bind
> echo "0000:06:00.1" >
/sys/bus/pci/devices/0000\:06\:00.1/driver/unbind
> echo "0000:06:00.1" > /sys/bus/pci/drivers/pci-stub/bind
> echo "0000:0c:00.0" >
/sys/bus/pci/devices/0000\:0c\:00.0/driver/unbind
> echo "0000:0c:00.0" > /sys/bus/pci/drivers/pci-stub/bind
>
> qemu-system-i386 -enable-kvm \
>                  -m 2048 \
>                  -cpu host \
>                  -machine pc-q35-1.6 \
>                  -drive file=/media/virtual/krym.qcow2,if=virtio \       
>                  -net none \
>                  -device pci-assign,host=06:00.1,rombar=0 \
>                  -device pci-assign,host=05:00.0,rombar=0 \
>                  -device pci-assign,host=06:00.0,rombar=0 \
>                  -device pci-assign,host=05:00.1,rombar=0 \
>                  -device pci-assign,host=0c:00.0,rombar=0 \
>                  -cdrom /media/virtual/asg-9.107-33.1.iso \
>                  -boot c \
>                  -vnc :0 \                                             
>                  -runas virtual \
>                  -monitor
unix:/media/virtual/control/krym/monitor,server,nowait \
>                  -rtc base=localtime,clock=host,driftfix=none
>
> This starts qemu and pci passthrough works nicely.
>
> Now, if I try to do the same thing through libvirt, virt-install
doesn't like the quad ethernet device pci passthrough and I get the error
below:
>
> virt-install --name=krym --ram=2048 --vcpus=1 --cpu=host --import
--os-type=linux --disk path=/media/virtual/krym.qcow2,format=qcow2,bus=virtio
--nonetworks --graphics vnc,port=5900 --virt-type=kvm --machine=q35 --accelerate
--host-device=pci_0000_06_00_1 --host-device=pci_0000_05_00_0
--host-device=pci_0000_06_00_0 --host-device=pci_0000_05_00_1
--host-device=pci_0000_0c_00_0
>
> Starting install...
> ERROR    Requested operation is not valid: PCI device 0000:06:00.1 is not
assignable
> Domain installation does not appear to have been successful.
> If it was, you can restart your domain by running:
>   virsh --connect qemu:///system start krym
> otherwise, please restart your installation.
More information is needed to make a guess at the source of the problem:

1) What distro/version of Linux are you running.
2) What version of libvirt?
3) After the failed virt-install command, if the new domain is still
defined, can you provide the output of:

    virsh dumpxml $domain

(if virt-install is automatically undefining the failed domain, then you
can get the xml for the defined domain by appending "--print-xml" to
the
virt-install commandline)

4) Also, send the last several lines of
/var/log/libvirt/qemu/$domain.log, in particular the qemu commandline
that was executed to start the domain, and any subsequent error messages.

(Just one possibility - since you are successfully using the q35
machinetype, it's highly likely that your OS is new enough that it
supports vfio for device passthrough; vfio is more restrictive about
what devices can be assigned to a guest, sometimes requiring seemingly
unrelated devices to be assigned to the same guest, and the most recent
libvirts will prefer vfio over legacy kvm device assignment when
available. So it's possible that your manual running of qemu-kvm works
because you are forcing use of legacy kvm assignment, and when libvirt
attempts to use vfio it is failing. This will be made apparent by the
information I requested above.

>
>
> If I try to use just one NIC (any NIC, as long as it's not the quad
NIC), things do work:
>
> virt-install --name=krym --ram=2048 --vcpus=1 --cpu=host --import
--os-type=linux --disk path=/media/virtual/krym.qcow2,format=qcow2,bus=virtio
--nonetworks --graphics vnc,port=5900 --virt-type=kvm --machine=q35 --accelerate
--host-device=pci_0000_0c_00_0
>
> Starting install...
> Creating domain...                                                         
|    0 B  00:00:01
> Cannot open display: 
> Run 'virt-viewer --help' to see a full list of available command
line options
> Domain creation completed. You can restart your domain by running:
>   virsh --connect qemu:///system start krym
>
>
>
> So I guess libvirt has some problem with the intel quad NIC.
>
> I will appreciate any help.
>
> Thanks,
> IvanK.
>
> _______________________________________________
> libvirt-users mailing list
> libvirt-users@redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users
>
>

Apparently Analagous Threads

Search for more seemingly similar threads

libvirt users - Jan 2014 - Re: intel quad gigabit nic and pci passthrough

[libvirt-users] intel quad gigabit nic and pci passthrough

Re: [libvirt-users] intel quad gigabit nic and pci passthrough

Re: [libvirt-users] intel quad gigabit nic and pci passthrough

Apparently Analagous Threads