Masroor Vettuparambil
2008-Oct-08 00:33 UTC
[Xen-devel] pci pass-through failure on xen 3.3.0
Hi all, I downloaded the xen 3.3.0 and now testing the pci pass-through with the NIC. Using late binding, the pci slot 0000:08:01.0 is assigned to pciback. But pci related xm commands are not working. For eg: linux-242:/home/xen-3.3.0 # xm pci-list-assignable-devices Unexpected error: <type ''exceptions.OSError''> Please report to xen-devel@lists.xensource.com Traceback (most recent call last): File "/usr/sbin/xm", line 10, in <module> main.main(sys.argv) File "//usr/lib64/python/xen/xm/main.py", line 2883, in main _, rc = _run_cmd(cmd, cmd_name, args) File "//usr/lib64/python/xen/xm/main.py", line 2907, in _run_cmd return True, cmd(args) File "//usr/lib64/python/xen/xm/main.py", line 2202, in xm_pci_list_assignable_devices devs_list = check_FLR_capability(dev_list) File "/usr/lib64/python2.5/xen/util/pci.py", line 288, in check_FLR_capability coassigned_pci_list = dev.find_coassigned_devices(True) File "/usr/lib64/python2.5/xen/util/pci.py", line 461, in find_coassigned_devices dev = self.find_the_uppermost_pci_bridge() File "/usr/lib64/python2.5/xen/util/pci.py", line 424, in find_the_uppermost_pci_bridge dev_parent = PciDevice(dom, b, d, f) File "/usr/lib64/python2.5/xen/util/pci.py", line 382, in __init__ self.detect_dev_info() File "/usr/lib64/python2.5/xen/util/pci.py", line 604, in detect_dev_info class_dev = self.pci_conf_read16(PCI_CLASS_DEVICE) File "/usr/lib64/python2.5/xen/util/pci.py", line 567, in pci_conf_read16 fd = os.open(self.cfg_space_path, os.O_RDONLY) OSError: [Errno 2] No such file or directory: ''/sys/bus/pci/devices/0000:06:00.0/config'' I checked the above python scripts. The function find_parent() in file util/pci.py, tries to find the parent device from the file link ../../../devices/pci0000:06/0000:06:01.0/0000:08:01.0 When it sees pci0000:06, it tries to read the config space of 0000:06:00.0 which doesn''t exist. This causes the above error. Lspci tree is, linux-242:/home/xen-3.3.0 # lspci -t -+-[0000:06]-+-01.0-[0000:08]----01.0 | +-01.1 | +-02.0-[0000:07]-- | \-02.1 \-[0000:00]-+-00.0 +-01.0 +-01.1 +-06.0 +-07.0 +-08.0 +-09.0-[0000:05]----07.0 +-0b.0-[0000:04]----00.0 +-0c.0-[0000:03]----00.0 +-0d.0-[0000:02]-- +-0e.0-[0000:01]-- +-18.0 +-18.1 +-18.2 +-18.3 +-19.0 +-19.1 +-19.2 \-19.3 Thanks, Masroor _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Masroor, Looks there is an incorrect assumption. Could you please help to test the attached patch? If it doesn't work, can you attached your output of "lspci -tv" and "lspci -xxx -vvv"? Thanks! -- Dexuan ________________________________ From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Masroor Vettuparambil Sent: 2008年10月8日 8:33 To: xen-devel@lists.xensource.com Subject: [Xen-devel] pci pass-through failure on xen 3.3.0 Hi all, I downloaded the xen 3.3.0 and now testing the pci pass-through with the NIC. Using late binding, the pci slot 0000:08:01.0 is assigned to pciback. But pci related xm commands are not working. For eg: linux-242:/home/xen-3.3.0 # xm pci-list-assignable-devices Unexpected error: <type 'exceptions.OSError'> Please report to xen-devel@lists.xensource.com Traceback (most recent call last): File "/usr/sbin/xm", line 10, in <module> main.main(sys.argv) File "//usr/lib64/python/xen/xm/main.py", line 2883, in main _, rc = _run_cmd(cmd, cmd_name, args) File "//usr/lib64/python/xen/xm/main.py", line 2907, in _run_cmd return True, cmd(args) File "//usr/lib64/python/xen/xm/main.py", line 2202, in xm_pci_list_assignable_devices devs_list = check_FLR_capability(dev_list) File "/usr/lib64/python2.5/xen/util/pci.py", line 288, in check_FLR_capability coassigned_pci_list = dev.find_coassigned_devices(True) File "/usr/lib64/python2.5/xen/util/pci.py", line 461, in find_coassigned_devices dev = self.find_the_uppermost_pci_bridge() File "/usr/lib64/python2.5/xen/util/pci.py", line 424, in find_the_uppermost_pci_bridge dev_parent = PciDevice(dom, b, d, f) File "/usr/lib64/python2.5/xen/util/pci.py", line 382, in __init__ self.detect_dev_info() File "/usr/lib64/python2.5/xen/util/pci.py", line 604, in detect_dev_info class_dev = self.pci_conf_read16(PCI_CLASS_DEVICE) File "/usr/lib64/python2.5/xen/util/pci.py", line 567, in pci_conf_read16 fd = os.open(self.cfg_space_path, os.O_RDONLY) OSError: [Errno 2] No such file or directory: '/sys/bus/pci/devices/0000:06:00.0/config' I checked the above python scripts. The function find_parent() in file util/pci.py, tries to find the parent device from the file link ../../../devices/pci0000:06/0000:06:01.0/0000:08:01.0 When it sees pci0000:06, it tries to read the config space of 0000:06:00.0 which doesn’t exist. This causes the above error. Lspci tree is, linux-242:/home/xen-3.3.0 # lspci -t -+-[0000:06]-+-01.0-[0000:08]----01.0 | +-01.1 | +-02.0-[0000:07]-- | \-02.1 \-[0000:00]-+-00.0 +-01.0 +-01.1 +-06.0 +-07.0 +-08.0 +-09.0-[0000:05]----07.0 +-0b.0-[0000:04]----00.0 +-0c.0-[0000:03]----00.0 +-0d.0-[0000:02]-- +-0e.0-[0000:01]-- +-18.0 +-18.1 +-18.2 +-18.3 +-19.0 +-19.1 +-19.2 \-19.3 Thanks, Masroor _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Masroor, BTW, you can simply apply the patch directly: login the target host, cd /usr/lib64/python/xen, patch -p4 < fix_find_parent.patch; xend restart. Thanks, -- Dexuan ________________________________ From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Cui, Dexuan Sent: 2008年10月8日 17:16 To: 'Masroor Vettuparambil'; 'xen-devel@lists.xensource.com' Subject: [Xen-devel] RE: pci pass-through failure on xen 3.3.0 Hi Masroor, Looks there is an incorrect assumption. Could you please help to test the attached patch? If it doesn't work, can you attached your output of "lspci -tv" and "lspci -xxx -vvv"? Thanks! -- Dexuan ________________________________ From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Masroor Vettuparambil Sent: 2008年10月8日 8:33 To: xen-devel@lists.xensource.com Subject: [Xen-devel] pci pass-through failure on xen 3.3.0 Hi all, I downloaded the xen 3.3.0 and now testing the pci pass-through with the NIC. Using late binding, the pci slot 0000:08:01.0 is assigned to pciback. But pci related xm commands are not working. For eg: linux-242:/home/xen-3.3.0 # xm pci-list-assignable-devices Unexpected error: <type 'exceptions.OSError'> Please report to xen-devel@lists.xensource.com Traceback (most recent call last): File "/usr/sbin/xm", line 10, in <module> main.main(sys.argv) File "//usr/lib64/python/xen/xm/main.py", line 2883, in main _, rc = _run_cmd(cmd, cmd_name, args) File "//usr/lib64/python/xen/xm/main.py", line 2907, in _run_cmd return True, cmd(args) File "//usr/lib64/python/xen/xm/main.py", line 2202, in xm_pci_list_assignable_devices devs_list = check_FLR_capability(dev_list) File "/usr/lib64/python2.5/xen/util/pci.py", line 288, in check_FLR_capability coassigned_pci_list = dev.find_coassigned_devices(True) File "/usr/lib64/python2.5/xen/util/pci.py", line 461, in find_coassigned_devices dev = self.find_the_uppermost_pci_bridge() File "/usr/lib64/python2.5/xen/util/pci.py", line 424, in find_the_uppermost_pci_bridge dev_parent = PciDevice(dom, b, d, f) File "/usr/lib64/python2.5/xen/util/pci.py", line 382, in __init__ self.detect_dev_info() File "/usr/lib64/python2.5/xen/util/pci.py", line 604, in detect_dev_info class_dev = self.pci_conf_read16(PCI_CLASS_DEVICE) File "/usr/lib64/python2.5/xen/util/pci.py", line 567, in pci_conf_read16 fd = os.open(self.cfg_space_path, os.O_RDONLY) OSError: [Errno 2] No such file or directory: '/sys/bus/pci/devices/0000:06:00.0/config' I checked the above python scripts. The function find_parent() in file util/pci.py, tries to find the parent device from the file link ../../../devices/pci0000:06/0000:06:01.0/0000:08:01.0 When it sees pci0000:06, it tries to read the config space of 0000:06:00.0 which doesn’t exist. This causes the above error. Lspci tree is, linux-242:/home/xen-3.3.0 # lspci -t -+-[0000:06]-+-01.0-[0000:08]----01.0 | +-01.1 | +-02.0-[0000:07]-- | \-02.1 \-[0000:00]-+-00.0 +-01.0 +-01.1 +-06.0 +-07.0 +-08.0 +-09.0-[0000:05]----07.0 +-0b.0-[0000:04]----00.0 +-0c.0-[0000:03]----00.0 +-0d.0-[0000:02]-- +-0e.0-[0000:01]-- +-18.0 +-18.1 +-18.2 +-18.3 +-19.0 +-19.1 +-19.2 \-19.3 Thanks, Masroor _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Masroor Vettuparambil
2008-Oct-10 01:42 UTC
[Xen-devel] RE: pci pass-through failure on xen 3.3.0
Hi Dexuan, Thanks for your patch. It looks like the below problem fixed. But now I ended up with “non-page aligned bar address” error. Xend log and lspci config space are given below, By the way, I could see in the script that all the pci devices behind the bridge should be co-assigned to the same guest. Does it mean that we could not export each device function on a multi-function card to separate guests? Xend log is, [2008-10-09 18:07:16 7867] ERROR (xmlrpclib2:178) Internal error handling xend.domain.device_configure Traceback (most recent call last): File "//usr/lib64/python/xen/util/xmlrpclib2.py", line 131, in _marshaled_dispatch response = self._dispatch(method, params) File "/usr/lib64/python2.5/SimpleXMLRPCServer.py", line 415, in _dispatch return func(*params) File "<string>", line 1, in <lambda> File "//usr/lib64/python/xen/xend/server/XMLRPCServer.py", line 55, in dispatch return getattr(info, fn)(*args) File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 821, in device_configure return self.pci_device_configure(dev_sxp) File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 722, in pci_device_configure self.device_create(dev_sxp) File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 637, in device_create self._createDevice(dev_type, dev_config_dict) File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 1869, in _createDevice return self.getDeviceController(deviceClass).createDevice(devConfig) File "//usr/lib64/python/xen/xend/server/DevController.py", line 94, in createDevice self.setupDevice(config) File "//usr/lib64/python/xen/xend/server/pciif.py", line 431, in setupDevice self.setupOneDevice(domain, bus, slot, func) File "//usr/lib64/python/xen/xend/server/pciif.py", line 289, in setupOneDevice raise VmError("pci: %: non-page-aligned MMIO BAR found." % dev.name) ValueError: unsupported format character \047:\047 (0x3a) at index 6 Config space dump of the NIC is, linux-242:# lspci -d 17d5: -v -xxx 08:01.0 Ethernet controller: S2io Inc. Xframe II 10Gbps Ethernet (rev 02) Subsystem: Hewlett-Packard Company Unknown device 1337 Flags: 66MHz, medium devsel, IRQ 9 Memory at ff5f8000 (64-bit, prefetchable) [disabled] [size=32K] Memory at ff400000 (64-bit, prefetchable) [disabled] [size=1M] Memory at ff5f7800 (64-bit, prefetchable) [disabled] [size=2K] Expansion ROM at fea00000 [disabled] [size=1M] Capabilities: [40] Message Signalled Interrupts: Mask+ 64bit+ Queue=0/5 Enable- Capabilities: [60] PCI-X non-bridge device Capabilities: [80] Vital Product Data Capabilities: [90] MSI-X: Enable- Mask- TabSize=64 Capabilities: [a0] Power Management version 3 00: d5 17 32 58 00 00 30 02 02 00 00 02 10 40 00 00 10: 0c 80 5f ff 00 00 00 00 0c 00 40 ff 00 00 00 00 20: 0c 78 5f ff 00 00 00 00 00 00 00 00 3c 10 37 13 30: 00 00 a0 fe 40 00 00 00 00 00 00 00 09 01 ff 01 40: 05 60 8a 01 00 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 07 80 73 10 08 08 e3 5f 00 00 00 00 00 00 00 00 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80: 03 90 fc 80 00 00 00 78 00 00 00 00 00 00 00 00 90: 11 a0 3f 00 04 00 00 00 04 04 00 00 00 00 00 00 a0: 01 00 23 7e 00 20 00 dc 00 00 00 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Thanks, Masroor From: Cui, Dexuan [mailto:dexuan.cui@intel.com] Sent: Wednesday, October 08, 2008 2:31 AM To: Masroor Vettuparambil; ''xen-devel@lists.xensource.com'' Subject: RE: pci pass-through failure on xen 3.3.0 Hi Masroor, BTW, you can simply apply the patch directly: login the target host, cd /usr/lib64/python/xen, patch -p4 < fix_find_parent.patch; xend restart. Thanks, -- Dexuan ________________________________ From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Cui, Dexuan Sent: 2008年10月8日 17:16 To: ''Masroor Vettuparambil''; ''xen-devel@lists.xensource.com'' Subject: [Xen-devel] RE: pci pass-through failure on xen 3.3.0 Hi Masroor, Looks there is an incorrect assumption. Could you please help to test the attached patch? If it doesn''t work, can you attached your output of "lspci -tv" and "lspci -xxx -vvv"? Thanks! -- Dexuan ________________________________ From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Masroor Vettuparambil Sent: 2008年10月8日 8:33 To: xen-devel@lists.xensource.com Subject: [Xen-devel] pci pass-through failure on xen 3.3.0 Hi all, I downloaded the xen 3.3.0 and now testing the pci pass-through with the NIC. Using late binding, the pci slot 0000:08:01.0 is assigned to pciback. But pci related xm commands are not working. For eg: linux-242:/home/xen-3.3.0 # xm pci-list-assignable-devices Unexpected error: <type ''exceptions.OSError''> Please report to xen-devel@lists.xensource.com Traceback (most recent call last): File "/usr/sbin/xm", line 10, in <module> main.main(sys.argv) File "//usr/lib64/python/xen/xm/main.py", line 2883, in main _, rc = _run_cmd(cmd, cmd_name, args) File "//usr/lib64/python/xen/xm/main.py", line 2907, in _run_cmd return True, cmd(args) File "//usr/lib64/python/xen/xm/main.py", line 2202, in xm_pci_list_assignable_devices devs_list = check_FLR_capability(dev_list) File "/usr/lib64/python2.5/xen/util/pci.py", line 288, in check_FLR_capability coassigned_pci_list = dev.find_coassigned_devices(True) File "/usr/lib64/python2.5/xen/util/pci.py", line 461, in find_coassigned_devices dev = self.find_the_uppermost_pci_bridge() File "/usr/lib64/python2.5/xen/util/pci.py", line 424, in find_the_uppermost_pci_bridge dev_parent = PciDevice(dom, b, d, f) File "/usr/lib64/python2.5/xen/util/pci.py", line 382, in __init__ self.detect_dev_info() File "/usr/lib64/python2.5/xen/util/pci.py", line 604, in detect_dev_info class_dev = self.pci_conf_read16(PCI_CLASS_DEVICE) File "/usr/lib64/python2.5/xen/util/pci.py", line 567, in pci_conf_read16 fd = os.open(self.cfg_space_path, os.O_RDONLY) OSError: [Errno 2] No such file or directory: ''/sys/bus/pci/devices/0000:06:00.0/config'' I checked the above python scripts. The function find_parent() in file util/pci.py, tries to find the parent device from the file link ../../../devices/pci0000:06/0000:06:01.0/0000:08:01.0 When it sees pci0000:06, it tries to read the config space of 0000:06:00.0 which doesn’t exist. This causes the above error. Lspci tree is, linux-242:/home/xen-3.3.0 # lspci -t -+-[0000:06]-+-01.0-[0000:08]----01.0 | +-01.1 | +-02.0-[0000:07]-- | \-02.1 \-[0000:00]-+-00.0 +-01.0 +-01.1 +-06.0 +-07.0 +-08.0 +-09.0-[0000:05]----07.0 +-0b.0-[0000:04]----00.0 +-0c.0-[0000:03]----00.0 +-0d.0-[0000:02]-- +-0e.0-[0000:01]-- +-18.0 +-18.1 +-18.2 +-18.3 +-19.0 +-19.1 +-19.2 \-19.3 Thanks, Masroor _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Masroor, Thanks for your testing! Glad to see an incorrect assumption is fixed here. For the non-page aligned BAR issue, it's expected: Your NIC has a non-page aligned BAR: Memory at ff5f7800 (64-bit, prefetchable) [disabled] [size=2K] Currently Xen disallows this kind of assignment due to some known issues... What's lucky is: Recently Yuji Shimada has posted a patch to resolve the case. Can you try the latest Dom0(changeset 693:2b5cc22ab406)? Thanks, -- Dexuan ________________________________ From: Masroor Vettuparambil [mailto:Masroor.Vettuparambil@neterion.com] Sent: 2008年10月10日 9:43 To: Cui, Dexuan; xen-devel@lists.xensource.com Subject: RE: pci pass-through failure on xen 3.3.0 Hi Dexuan, Thanks for your patch. It looks like the below problem fixed. But now I ended up with “non-page aligned bar address” error. Xend log and lspci config space are given below, By the way, I could see in the script that all the pci devices behind the bridge should be co-assigned to the same guest. Does it mean that we could not export each device function on a multi-function card to separate guests? Xend log is, [2008-10-09 18:07:16 7867] ERROR (xmlrpclib2:178) Internal error handling xend.domain.device_configure Traceback (most recent call last): File "//usr/lib64/python/xen/util/xmlrpclib2.py", line 131, in _marshaled_dispatch response = self._dispatch(method, params) File "/usr/lib64/python2.5/SimpleXMLRPCServer.py", line 415, in _dispatch return func(*params) File "<string>", line 1, in <lambda> File "//usr/lib64/python/xen/xend/server/XMLRPCServer.py", line 55, in dispatch return getattr(info, fn)(*args) File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 821, in device_configure return self.pci_device_configure(dev_sxp) File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 722, in pci_device_configure self.device_create(dev_sxp) File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 637, in device_create self._createDevice(dev_type, dev_config_dict) File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 1869, in _createDevice return self.getDeviceController(deviceClass).createDevice(devConfig) File "//usr/lib64/python/xen/xend/server/DevController.py", line 94, in createDevice self.setupDevice(config) File "//usr/lib64/python/xen/xend/server/pciif.py", line 431, in setupDevice self.setupOneDevice(domain, bus, slot, func) File "//usr/lib64/python/xen/xend/server/pciif.py", line 289, in setupOneDevice raise VmError("pci: %: non-page-aligned MMIO BAR found." % dev.name) ValueError: unsupported format character \047:\047 (0x3a) at index 6 Config space dump of the NIC is, linux-242:# lspci -d 17d5: -v -xxx 08:01.0 Ethernet controller: S2io Inc. Xframe II 10Gbps Ethernet (rev 02) Subsystem: Hewlett-Packard Company Unknown device 1337 Flags: 66MHz, medium devsel, IRQ 9 Memory at ff5f8000 (64-bit, prefetchable) [disabled] [size=32K] Memory at ff400000 (64-bit, prefetchable) [disabled] [size=1M] Memory at ff5f7800 (64-bit, prefetchable) [disabled] [size=2K] Expansion ROM at fea00000 [disabled] [size=1M] Capabilities: [40] Message Signalled Interrupts: Mask+ 64bit+ Queue=0/5 Enable- Capabilities: [60] PCI-X non-bridge device Capabilities: [80] Vital Product Data Capabilities: [90] MSI-X: Enable- Mask- TabSize=64 Capabilities: [a0] Power Management version 3 00: d5 17 32 58 00 00 30 02 02 00 00 02 10 40 00 00 10: 0c 80 5f ff 00 00 00 00 0c 00 40 ff 00 00 00 00 20: 0c 78 5f ff 00 00 00 00 00 00 00 00 3c 10 37 13 30: 00 00 a0 fe 40 00 00 00 00 00 00 00 09 01 ff 01 40: 05 60 8a 01 00 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 07 80 73 10 08 08 e3 5f 00 00 00 00 00 00 00 00 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80: 03 90 fc 80 00 00 00 78 00 00 00 00 00 00 00 00 90: 11 a0 3f 00 04 00 00 00 04 04 00 00 00 00 00 00 a0: 01 00 23 7e 00 20 00 dc 00 00 00 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Thanks, Masroor From: Cui, Dexuan [mailto:dexuan.cui@intel.com] Sent: Wednesday, October 08, 2008 2:31 AM To: Masroor Vettuparambil; 'xen-devel@lists.xensource.com' Subject: RE: pci pass-through failure on xen 3.3.0 Hi Masroor, BTW, you can simply apply the patch directly: login the target host, cd /usr/lib64/python/xen, patch -p4 < fix_find_parent.patch; xend restart. Thanks, -- Dexuan ________________________________ From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Cui, Dexuan Sent: 2008年10月8日 17:16 To: 'Masroor Vettuparambil'; 'xen-devel@lists.xensource.com' Subject: [Xen-devel] RE: pci pass-through failure on xen 3.3.0 Hi Masroor, Looks there is an incorrect assumption. Could you please help to test the attached patch? If it doesn't work, can you attached your output of "lspci -tv" and "lspci -xxx -vvv"? Thanks! -- Dexuan ________________________________ From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Masroor Vettuparambil Sent: 2008年10月8日 8:33 To: xen-devel@lists.xensource.com Subject: [Xen-devel] pci pass-through failure on xen 3.3.0 Hi all, I downloaded the xen 3.3.0 and now testing the pci pass-through with the NIC. Using late binding, the pci slot 0000:08:01.0 is assigned to pciback. But pci related xm commands are not working. For eg: linux-242:/home/xen-3.3.0 # xm pci-list-assignable-devices Unexpected error: <type 'exceptions.OSError'> Please report to xen-devel@lists.xensource.com Traceback (most recent call last): File "/usr/sbin/xm", line 10, in <module> main.main(sys.argv) File "//usr/lib64/python/xen/xm/main.py", line 2883, in main _, rc = _run_cmd(cmd, cmd_name, args) File "//usr/lib64/python/xen/xm/main.py", line 2907, in _run_cmd return True, cmd(args) File "//usr/lib64/python/xen/xm/main.py", line 2202, in xm_pci_list_assignable_devices devs_list = check_FLR_capability(dev_list) File "/usr/lib64/python2.5/xen/util/pci.py", line 288, in check_FLR_capability coassigned_pci_list = dev.find_coassigned_devices(True) File "/usr/lib64/python2.5/xen/util/pci.py", line 461, in find_coassigned_devices dev = self.find_the_uppermost_pci_bridge() File "/usr/lib64/python2.5/xen/util/pci.py", line 424, in find_the_uppermost_pci_bridge dev_parent = PciDevice(dom, b, d, f) File "/usr/lib64/python2.5/xen/util/pci.py", line 382, in __init__ self.detect_dev_info() File "/usr/lib64/python2.5/xen/util/pci.py", line 604, in detect_dev_info class_dev = self.pci_conf_read16(PCI_CLASS_DEVICE) File "/usr/lib64/python2.5/xen/util/pci.py", line 567, in pci_conf_read16 fd = os.open(self.cfg_space_path, os.O_RDONLY) OSError: [Errno 2] No such file or directory: '/sys/bus/pci/devices/0000:06:00.0/config' I checked the above python scripts. The function find_parent() in file util/pci.py, tries to find the parent device from the file link ../../../devices/pci0000:06/0000:06:01.0/0000:08:01.0 When it sees pci0000:06, it tries to read the config space of 0000:06:00.0 which doesn’t exist. This causes the above error. Lspci tree is, linux-242:/home/xen-3.3.0 # lspci -t -+-[0000:06]-+-01.0-[0000:08]----01.0 | +-01.1 | +-02.0-[0000:07]-- | \-02.1 \-[0000:00]-+-00.0 +-01.0 +-01.1 +-06.0 +-07.0 +-08.0 +-09.0-[0000:05]----07.0 +-0b.0-[0000:04]----00.0 +-0c.0-[0000:03]----00.0 +-0d.0-[0000:02]-- +-0e.0-[0000:01]-- +-18.0 +-18.1 +-18.2 +-18.3 +-19.0 +-19.1 +-19.2 \-19.3 Thanks, Masroor _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Masroor, There are many important bug fixes in the latest 3.3 tree. Maybe it's better to try the latest 3.3 code. For instance, you may need a small changeset 18381:24083a2dba7f to fix a format typo here. :-) For the co-assignment, if a device(including multi-function device) hasn't a proper standard FLR methods, we have to use the SecondaryBusReset as a FLR method, so we require the co-assignment. You can refer to http://xenbits.xensource.com/xen-3.3-testing.hg?rev/e61978c24d84 for details. Thanks, -- Dexuan ________________________________ From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Cui, Dexuan Sent: 2008年10月10日 10:01 To: 'Masroor Vettuparambil'; 'xen-devel@lists.xensource.com' Subject: [Xen-devel] RE: pci pass-through failure on xen 3.3.0 Hi Masroor, Thanks for your testing! Glad to see an incorrect assumption is fixed here. For the non-page aligned BAR issue, it's expected: Your NIC has a non-page aligned BAR: Memory at ff5f7800 (64-bit, prefetchable) [disabled] [size=2K] Currently Xen disallows this kind of assignment due to some known issues... What's lucky is: Recently Yuji Shimada has posted a patch to resolve the case. Can you try the latest Dom0(changeset 693:2b5cc22ab406)? Thanks, -- Dexuan ________________________________ From: Masroor Vettuparambil [mailto:Masroor.Vettuparambil@neterion.com] Sent: 2008年10月10日 9:43 To: Cui, Dexuan; xen-devel@lists.xensource.com Subject: RE: pci pass-through failure on xen 3.3.0 Hi Dexuan, Thanks for your patch. It looks like the below problem fixed. But now I ended up with “non-page aligned bar address” error. Xend log and lspci config space are given below, By the way, I could see in the script that all the pci devices behind the bridge should be co-assigned to the same guest. Does it mean that we could not export each device function on a multi-function card to separate guests? Xend log is, [2008-10-09 18:07:16 7867] ERROR (xmlrpclib2:178) Internal error handling xend.domain.device_configure Traceback (most recent call last): File "//usr/lib64/python/xen/util/xmlrpclib2.py", line 131, in _marshaled_dispatch response = self._dispatch(method, params) File "/usr/lib64/python2.5/SimpleXMLRPCServer.py", line 415, in _dispatch return func(*params) File "<string>", line 1, in <lambda> File "//usr/lib64/python/xen/xend/server/XMLRPCServer.py", line 55, in dispatch return getattr(info, fn)(*args) File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 821, in device_configure return self.pci_device_configure(dev_sxp) File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 722, in pci_device_configure self.device_create(dev_sxp) File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 637, in device_create self._createDevice(dev_type, dev_config_dict) File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 1869, in _createDevice return self.getDeviceController(deviceClass).createDevice(devConfig) File "//usr/lib64/python/xen/xend/server/DevController.py", line 94, in createDevice self.setupDevice(config) File "//usr/lib64/python/xen/xend/server/pciif.py", line 431, in setupDevice self.setupOneDevice(domain, bus, slot, func) File "//usr/lib64/python/xen/xend/server/pciif.py", line 289, in setupOneDevice raise VmError("pci: %: non-page-aligned MMIO BAR found." % dev.name) ValueError: unsupported format character \047:\047 (0x3a) at index 6 Config space dump of the NIC is, linux-242:# lspci -d 17d5: -v -xxx 08:01.0 Ethernet controller: S2io Inc. Xframe II 10Gbps Ethernet (rev 02) Subsystem: Hewlett-Packard Company Unknown device 1337 Flags: 66MHz, medium devsel, IRQ 9 Memory at ff5f8000 (64-bit, prefetchable) [disabled] [size=32K] Memory at ff400000 (64-bit, prefetchable) [disabled] [size=1M] Memory at ff5f7800 (64-bit, prefetchable) [disabled] [size=2K] Expansion ROM at fea00000 [disabled] [size=1M] Capabilities: [40] Message Signalled Interrupts: Mask+ 64bit+ Queue=0/5 Enable- Capabilities: [60] PCI-X non-bridge device Capabilities: [80] Vital Product Data Capabilities: [90] MSI-X: Enable- Mask- TabSize=64 Capabilities: [a0] Power Management version 3 00: d5 17 32 58 00 00 30 02 02 00 00 02 10 40 00 00 10: 0c 80 5f ff 00 00 00 00 0c 00 40 ff 00 00 00 00 20: 0c 78 5f ff 00 00 00 00 00 00 00 00 3c 10 37 13 30: 00 00 a0 fe 40 00 00 00 00 00 00 00 09 01 ff 01 40: 05 60 8a 01 00 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 07 80 73 10 08 08 e3 5f 00 00 00 00 00 00 00 00 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80: 03 90 fc 80 00 00 00 78 00 00 00 00 00 00 00 00 90: 11 a0 3f 00 04 00 00 00 04 04 00 00 00 00 00 00 a0: 01 00 23 7e 00 20 00 dc 00 00 00 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Thanks, Masroor From: Cui, Dexuan [mailto:dexuan.cui@intel.com] Sent: Wednesday, October 08, 2008 2:31 AM To: Masroor Vettuparambil; 'xen-devel@lists.xensource.com' Subject: RE: pci pass-through failure on xen 3.3.0 Hi Masroor, BTW, you can simply apply the patch directly: login the target host, cd /usr/lib64/python/xen, patch -p4 < fix_find_parent.patch; xend restart. Thanks, -- Dexuan ________________________________ From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Cui, Dexuan Sent: 2008年10月8日 17:16 To: 'Masroor Vettuparambil'; 'xen-devel@lists.xensource.com' Subject: [Xen-devel] RE: pci pass-through failure on xen 3.3.0 Hi Masroor, Looks there is an incorrect assumption. Could you please help to test the attached patch? If it doesn't work, can you attached your output of "lspci -tv" and "lspci -xxx -vvv"? Thanks! -- Dexuan ________________________________ From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Masroor Vettuparambil Sent: 2008年10月8日 8:33 To: xen-devel@lists.xensource.com Subject: [Xen-devel] pci pass-through failure on xen 3.3.0 Hi all, I downloaded the xen 3.3.0 and now testing the pci pass-through with the NIC. Using late binding, the pci slot 0000:08:01.0 is assigned to pciback. But pci related xm commands are not working. For eg: linux-242:/home/xen-3.3.0 # xm pci-list-assignable-devices Unexpected error: <type 'exceptions.OSError'> Please report to xen-devel@lists.xensource.com Traceback (most recent call last): File "/usr/sbin/xm", line 10, in <module> main.main(sys.argv) File "//usr/lib64/python/xen/xm/main.py", line 2883, in main _, rc = _run_cmd(cmd, cmd_name, args) File "//usr/lib64/python/xen/xm/main.py", line 2907, in _run_cmd return True, cmd(args) File "//usr/lib64/python/xen/xm/main.py", line 2202, in xm_pci_list_assignable_devices devs_list = check_FLR_capability(dev_list) File "/usr/lib64/python2.5/xen/util/pci.py", line 288, in check_FLR_capability coassigned_pci_list = dev.find_coassigned_devices(True) File "/usr/lib64/python2.5/xen/util/pci.py", line 461, in find_coassigned_devices dev = self.find_the_uppermost_pci_bridge() File "/usr/lib64/python2.5/xen/util/pci.py", line 424, in find_the_uppermost_pci_bridge dev_parent = PciDevice(dom, b, d, f) File "/usr/lib64/python2.5/xen/util/pci.py", line 382, in __init__ self.detect_dev_info() File "/usr/lib64/python2.5/xen/util/pci.py", line 604, in detect_dev_info class_dev = self.pci_conf_read16(PCI_CLASS_DEVICE) File "/usr/lib64/python2.5/xen/util/pci.py", line 567, in pci_conf_read16 fd = os.open(self.cfg_space_path, os.O_RDONLY) OSError: [Errno 2] No such file or directory: '/sys/bus/pci/devices/0000:06:00.0/config' I checked the above python scripts. The function find_parent() in file util/pci.py, tries to find the parent device from the file link ../../../devices/pci0000:06/0000:06:01.0/0000:08:01.0 When it sees pci0000:06, it tries to read the config space of 0000:06:00.0 which doesn’t exist. This causes the above error. Lspci tree is, linux-242:/home/xen-3.3.0 # lspci -t -+-[0000:06]-+-01.0-[0000:08]----01.0 | +-01.1 | +-02.0-[0000:07]-- | \-02.1 \-[0000:00]-+-00.0 +-01.0 +-01.1 +-06.0 +-07.0 +-08.0 +-09.0-[0000:05]----07.0 +-0b.0-[0000:04]----00.0 +-0c.0-[0000:03]----00.0 +-0d.0-[0000:02]-- +-0e.0-[0000:01]-- +-18.0 +-18.1 +-18.2 +-18.3 +-19.0 +-19.1 +-19.2 \-19.3 Thanks, Masroor _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Masroor Vettuparambil
2008-Oct-10 05:37 UTC
[Xen-devel] RE: pci pass-through failure on xen 3.3.0
Thanks for pointing it. I will try the latest dom0 and xen. ~Masroor ________________________________ From: Cui, Dexuan [mailto:dexuan.cui@intel.com] Sent: Thursday, October 09, 2008 7:47 PM To: Masroor Vettuparambil; ''xen-devel@lists.xensource.com'' Subject: RE: pci pass-through failure on xen 3.3.0 Hi Masroor, There are many important bug fixes in the latest 3.3 tree. Maybe it''s better to try the latest 3.3 code. For instance, you may need a small changeset 18381:24083a2dba7f to fix a format typo here. :-) For the co-assignment, if a device(includingmulti-function device) hasn''t a proper standard FLR methods, we have to use the SecondaryBusReset as a FLR method, so we require the co-assignment. You can refer to http://xenbits.xensource.com/xen-3.3-testing.hg?rev/e61978c24d84for details. Thanks, -- Dexuan ________________________________ From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Cui, Dexuan Sent: 2008年10月10日 10:01 To: ''Masroor Vettuparambil''; ''xen-devel@lists.xensource.com'' Subject: [Xen-devel] RE: pci pass-through failure on xen 3.3.0 Hi Masroor, Thanks for your testing! Glad to see an incorrect assumption is fixed here. For the non-page aligned BAR issue, it''s expected: Your NIC has a non-page aligned BAR: Memory at ff5f7800 (64-bit, prefetchable) [disabled] [size=2K] Currently Xen disallows this kind of assignment due to some known issues... What''s lucky is: Recently Yuji Shimada has posted a patch to resolve the case. Can you try the latest Dom0(changeset 693:2b5cc22ab406)? Thanks, -- Dexuan ________________________________ From: Masroor Vettuparambil [mailto:Masroor.Vettuparambil@neterion.com] Sent: 2008年10月10日 9:43 To: Cui, Dexuan; xen-devel@lists.xensource.com Subject: RE: pci pass-through failure on xen 3.3.0 Hi Dexuan, Thanks for your patch. It looks like the below problem fixed. But now I ended up with “non-page aligned bar address” error. Xend log and lspci config space are given below, By the way, I could see in the script that all the pci devices behind the bridge should be co-assigned to the same guest. Does it mean that we could not export each device function on a multi-function card to separate guests? Xend log is, [2008-10-09 18:07:16 7867] ERROR (xmlrpclib2:178) Internal error handling xend.domain.device_configure Traceback (most recent call last): File "//usr/lib64/python/xen/util/xmlrpclib2.py", line 131, in _marshaled_dispatch response = self._dispatch(method, params) File "/usr/lib64/python2.5/SimpleXMLRPCServer.py", line 415, in _dispatch return func(*params) File "<string>", line 1, in <lambda> File "//usr/lib64/python/xen/xend/server/XMLRPCServer.py", line 55, in dispatch return getattr(info, fn)(*args) File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 821, in device_configure return self.pci_device_configure(dev_sxp) File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 722, in pci_device_configure self.device_create(dev_sxp) File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 637, in device_create self._createDevice(dev_type, dev_config_dict) File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 1869, in _createDevice return self.getDeviceController(deviceClass).createDevice(devConfig) File "//usr/lib64/python/xen/xend/server/DevController.py", line 94, in createDevice self.setupDevice(config) File "//usr/lib64/python/xen/xend/server/pciif.py", line 431, in setupDevice self.setupOneDevice(domain, bus, slot, func) File "//usr/lib64/python/xen/xend/server/pciif.py", line 289, in setupOneDevice raise VmError("pci: %: non-page-aligned MMIO BAR found." % dev.name) ValueError: unsupported format character \047:\047 (0x3a) at index 6 Config space dump of the NIC is, linux-242:# lspci -d 17d5: -v -xxx 08:01.0 Ethernet controller: S2io Inc. Xframe II 10Gbps Ethernet (rev 02) Subsystem: Hewlett-Packard Company Unknown device 1337 Flags: 66MHz, medium devsel, IRQ 9 Memory at ff5f8000 (64-bit, prefetchable) [disabled] [size=32K] Memory at ff400000 (64-bit, prefetchable) [disabled] [size=1M] Memory at ff5f7800 (64-bit, prefetchable) [disabled] [size=2K] Expansion ROM at fea00000 [disabled] [size=1M] Capabilities: [40] Message Signalled Interrupts: Mask+ 64bit+ Queue=0/5 Enable- Capabilities: [60] PCI-X non-bridge device Capabilities: [80] Vital Product Data Capabilities: [90] MSI-X: Enable- Mask- TabSize=64 Capabilities: [a0] Power Management version 3 00: d5 17 32 58 00 00 30 02 02 00 00 02 10 40 00 00 10: 0c 80 5f ff 00 00 00 00 0c 00 40 ff 00 00 00 00 20: 0c 78 5f ff 00 00 00 00 00 00 00 00 3c 10 37 13 30: 00 00 a0 fe 40 00 00 00 00 00 00 00 09 01 ff 01 40: 05 60 8a 01 00 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 07 80 73 10 08 08 e3 5f 00 00 00 00 00 00 00 00 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80: 03 90 fc 80 00 00 00 78 00 00 00 00 00 00 00 00 90: 11 a0 3f 00 04 00 00 00 04 04 00 00 00 00 00 00 a0: 01 00 23 7e 00 20 00 dc 00 00 00 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Thanks, Masroor From: Cui, Dexuan [mailto:dexuan.cui@intel.com] Sent: Wednesday, October 08, 2008 2:31 AM To: Masroor Vettuparambil; ''xen-devel@lists.xensource.com'' Subject: RE: pci pass-through failure on xen 3.3.0 Hi Masroor, BTW, you can simply apply the patch directly: login the target host, cd /usr/lib64/python/xen, patch -p4 < fix_find_parent.patch; xend restart. Thanks, -- Dexuan ________________________________ From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Cui, Dexuan Sent: 2008年10月8日 17:16 To: ''Masroor Vettuparambil''; ''xen-devel@lists.xensource.com'' Subject: [Xen-devel] RE: pci pass-through failure on xen 3.3.0 Hi Masroor, Looks there is an incorrect assumption. Could you please help to test the attached patch? If it doesn''t work, can you attached your output of "lspci -tv" and "lspci -xxx -vvv"? Thanks! -- Dexuan ________________________________ From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Masroor Vettuparambil Sent: 2008年10月8日 8:33 To: xen-devel@lists.xensource.com Subject: [Xen-devel] pci pass-through failure on xen 3.3.0 Hi all, I downloaded the xen 3.3.0 and now testing the pci pass-through with the NIC. Using late binding, the pci slot 0000:08:01.0 is assigned to pciback. But pci related xm commands are not working. For eg: linux-242:/home/xen-3.3.0 # xm pci-list-assignable-devices Unexpected error: <type ''exceptions.OSError''> Please report to xen-devel@lists.xensource.com Traceback (most recent call last): File "/usr/sbin/xm", line 10, in <module> main.main(sys.argv) File "//usr/lib64/python/xen/xm/main.py", line 2883, in main _, rc = _run_cmd(cmd, cmd_name, args) File "//usr/lib64/python/xen/xm/main.py", line 2907, in _run_cmd return True, cmd(args) File "//usr/lib64/python/xen/xm/main.py", line 2202, in xm_pci_list_assignable_devices devs_list = check_FLR_capability(dev_list) File "/usr/lib64/python2.5/xen/util/pci.py", line 288, in check_FLR_capability coassigned_pci_list = dev.find_coassigned_devices(True) File "/usr/lib64/python2.5/xen/util/pci.py", line 461, in find_coassigned_devices dev = self.find_the_uppermost_pci_bridge() File "/usr/lib64/python2.5/xen/util/pci.py", line 424, in find_the_uppermost_pci_bridge dev_parent = PciDevice(dom, b, d, f) File "/usr/lib64/python2.5/xen/util/pci.py", line 382, in __init__ self.detect_dev_info() File "/usr/lib64/python2.5/xen/util/pci.py", line 604, in detect_dev_info class_dev = self.pci_conf_read16(PCI_CLASS_DEVICE) File "/usr/lib64/python2.5/xen/util/pci.py", line 567, in pci_conf_read16 fd = os.open(self.cfg_space_path, os.O_RDONLY) OSError: [Errno 2] No such file or directory: ''/sys/bus/pci/devices/0000:06:00.0/config'' I checked the above python scripts. The function find_parent() in file util/pci.py, tries to find the parent device from the file link ../../../devices/pci0000:06/0000:06:01.0/0000:08:01.0 When it sees pci0000:06, it tries to read the config space of 0000:06:00.0 which doesn’t exist. This causes the above error. Lspci tree is, linux-242:/home/xen-3.3.0 # lspci -t -+-[0000:06]-+-01.0-[0000:08]----01.0 | +-01.1 | +-02.0-[0000:07]-- | \-02.1 \-[0000:00]-+-00.0 +-01.0 +-01.1 +-06.0 +-07.0 +-08.0 +-09.0-[0000:05]----07.0 +-0b.0-[0000:04]----00.0 +-0c.0-[0000:03]----00.0 +-0d.0-[0000:02]-- +-0e.0-[0000:01]-- +-18.0 +-18.1 +-18.2 +-18.3 +-19.0 +-19.1 +-19.2 \-19.3 Thanks, Masroor _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel