On 3/22/23 04:23, nospam at godawa.de wrote:> Jim Fehlig schrieb: >> >> What is the libvirt version? > > It's the "latest and greatest" I get from this source: > > [root at xen1 ~]# virsh --version > 6.6.0 > > [root at xen1 ~]# libvirtd --version > libvirtd (libvirt) 6.6.0 > > [root at xengfs1f ~]# yum list | grep libvirt > libvirt.x86_64?????????????????????????? 1:6.6.0-6.xen415.el7 @centos-virt-xen-415 > libvirt-bash-completion.x86_64?????????? 1:6.6.0-6.xen415.el7 @centos-virt-xen-415 > libvirt-client.x86_64??????????????????? 1:6.6.0-6.xen415.el7 @centos-virt-xen-415 > libvirt-daemon.x86_64??????????????????? 1:6.6.0-6.xen415.el7 @centos-virt-xen-415That's an old libvirt, but there hasn't been a lot of changes to the PCI passthrough code. One notable change that came with libvirt 6.8.0 https://gitlab.com/libvirt/libvirt/-/commit/9d15647dcb96831c93ac8c1d67c47265b5ed9072 However, that wouldn't be needed unless you were using the 'permissive' setting in your xl config.>> I know it's not complete, but https://libvirt.org/formatdomain.html makes an >> attempt to identify hypervisor support for the various configuration settings. > > I will "recheck" it, but I don't see anything new, that might be missing. > > >> I've used both configurations successfully. Has the sriov virtual function >> been unbound from the native driver and bound to xen-pciback? > > Should be, otherwise it would not work with XL? > > >> You can check which driver is in use with something like >> >> ls /sys/bus/pci/devices/0000\:81\:02.6/driver > > I can't see the exact definition, but again, same output when starting with xl > or virsh: > > [root at xen1 ~]# ls /sys/bus/pci/devices/0000\:81\:02.6/driver > 0000:81:02.0? 0000:81:02.7? 0000:81:03.6? 0000:81:04.5? 0000:81:05.4 > 0000:81:06.5? 0000:81:07.4? 0000:81:08.3? 0000:81:09.2?????? module ?slots > 0000:81:02.1? 0000:81:03.0? 0000:81:03.7? 0000:81:04.6? 0000:81:05.5 > 0000:81:06.6? 0000:81:07.5? 0000:81:08.4? 0000:81:09.3?????? new_id ?uevent > 0000:81:02.2? 0000:81:03.1? 0000:81:04.0? 0000:81:04.7? 0000:81:06.0 > 0000:81:06.7? 0000:81:07.6? 0000:81:08.5? 0000:81:09.4?????? new_slot ?unbind > 0000:81:02.3? 0000:81:03.2? 0000:81:04.1? 0000:81:05.0? 0000:81:06.1 > 0000:81:07.0? 0000:81:07.7? 0000:81:08.6? 0000:81:09.5?????? permissive > 0000:81:02.4? 0000:81:03.3? 0000:81:04.2? 0000:81:05.1? 0000:81:06.2 > 0000:81:07.1? 0000:81:08.0? 0000:81:08.7? bind?????????????? quirks > 0000:81:02.5? 0000:81:03.4? 0000:81:04.3? 0000:81:05.2? 0000:81:06.3 > 0000:81:07.2? 0000:81:08.1? 0000:81:09.0? irq_handlers?????? remove_id > 0000:81:02.6? 0000:81:03.5? 0000:81:04.4? 0000:81:05.3? 0000:81:06.4 > 0000:81:07.3? 0000:81:08.2? 0000:81:09.1? irq_handler_state? remove_slot > > [root at xen1 ~]# cat /sys/bus/pci/devices/0000\:81\:02.6/device > 0x154c > [root at xen1 ~]# cat /sys/bus/pci/devices/0000\:81\:02.6/vendor > 0x8086 > [root at xen1 ~]# cat /sys/bus/pci/devices/0000\:81\:02.6/subsystem_vendor > 0x8086 > > [root at xen1 ~]# ls /sys/bus/pci/devices/0000\:81\:02.6/driver/module/drivers/ > pci:pciback? xen-backend:xen-pciback > > [root at xen1 ~]# ls /sys/bus/pci/devices/0000\:81\:02.6/driver/0000:81:02.6 > broken_parity_status? consistent_dma_mask_bits? dma_mask_bits??? enable > local_cpus? numa_node? reset????? resource0_wc? subsystem ?uevent > class???????????????? d3cold_allowed??????????? driver?????????? irq > modalias??? physfn???? resource?? resource3???? subsystem_device ?vendor > config??????????????? device??????????????????? driver_override local_cpulist > msi_bus???? power????? resource0? resource3_wc subsystem_vendor > > >> If bound to xen-pciback, it vf should also appear in the output of 'xl >> pci-assignable-list'. > > Before starting the VM, it's there: > > [root at xengfs1f ~]# xl pci-assignable-list | grep "0000:81:02.6" > 0000:81:02.6Ah, xen-pciback is already bound to the device, so no need for managed='yes' in your device config. The 'managed' attribute tells libvirt whether or not to manage attaching/detaching drivers to/from the device. If you've done that elsewhere, set managed='no', or simply drop the attribute since 'no' is the default. It's explained in the 'pci' attribute of the hostdev element https://libvirt.org/formatdomain.html#usb-pci-scsi-devices But I wouldn't expect that minor config optimization to fix your issue. I tried starting a VM with managed='yes' and the device already assigned to xen-pciback, but it still worked for me. The sriov vf was visible and usable in the VM. Are there any errors from xen-pciback in dom0, or any hints in the output of 'xl dmesg'? Also, it might be worth comparing the relevant nodes in the output of xenstore-ls between VM started with xl and libvirt. Regards, Jim
Hi Jim, thank you very much for your answers! Unfortunately I was busy with some other things, so couldn't look at it earlier.> That's an old libvirt, but there hasn't been a lot of changes to the PCI > passthrough code. One notable change that came with libvirt 6.8.0 > > https://gitlab.com/libvirt/libvirt/-/commit/9d15647dcb96831c93ac8c1d67c47265b5ed9072 > > However, that wouldn't be needed unless you were using the 'permissive' > setting in your xl config.I see, current version is somewhere at 9.x. But I'm not sure, if it's possible to upgade the servers to a new version of libvirt, I assume, that I also would have to upgrade Xen to the latest version.>>> I know it's not complete, but https://libvirt.org/formatdomain.html >>> makes an attempt to identify hypervisor support for the various >>> configuration settings.I found this: "if you are using a version of libvirt older than 0.9.11, you should use standard <hostdev> to assign the device to the guest instead of <interface type='hostdev'/>." I think that 6.6 is newer than 0.9.11? Or do I have 0.6.6 in real?> Ah, xen-pciback is already bound to the device, so no need for > managed='yes' in your device config. The 'managed' attribute tells > libvirt whether or not to manage attaching/detaching drivers to/from the > device. If you've done that elsewhere, set managed='no', or simply drop > the attribute since 'no' is the default. It's explained in the 'pci' > attribute of the hostdev element > > https://libvirt.org/formatdomain.html#usb-pci-scsi-devicesI set this to "no", as you expected it didn't changed anything.> Are there any errors from xen-pciback in dom0, or any hints in the > output of 'xl dmesg'? Also, it might be worth comparing the relevant > nodes in the output of xenstore-ls between VM started with xl and libvirt.I already had compared some informations between XL and libvirt, see my very first message. Unfortunately there are no errors in the logs, it's just not working. Any hints how to get further informations that might help debugging? Or could it just be a old bug in this version of libvirt? Thanks a lot, -- kind regards, Thorolf