I''ve asked a couple of these questions, already, but haven''t really had any response, so I thought I''d try, again, and try to provide a little more detail. I need to pass through a PCIe 4-port Serial card. I''ve done this successfully multiple times with different serial cards, but the current one, a PCIe Perle quad-port RJ45, is causing some problems. I believe the problem stems from the fact that the card is a PCI card and that Perle just built a PCIe-PCI bridge so that they could reuse an existing PCI design. The following three PCI devices are all on the same card: # lspci ... 01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge (rev aa) 02:00.0 Serial controller: Perle Systems Ltd Device 0331 02:00.1 Bridge: Perle Systems Ltd Device 0331 I''ve tried various combinations of hiding 01:00.0, 02:00.0, and/or 02:00.1 from dom0 and passing those through. First, if I try all three, I put the following in my dom0 kernel arguments: pciback.hide=(01:00.0)(02:00.0)(02:00.1) Before someone suggests that maybe I have my kernel arguments wrong, this is a SuSE kernel, so it is not pvops, and that is the correct syntax - I can add other PCI(e) devices on the same exact host (like a graphics card) and pass those through without any problem at all. So syntax is correct. After I boot with these arguments, I check the dmesg output for pciback messages: # dmesg|grep pciback [ 0.000000] Command line: root=/dev/local/dom0 resume=/dev/local/swap pciback.hide=(01:00.0)(02:00.0)(02:00.1) splash=silent quiet showopts [ 0.000000] Kernel command line: root=/dev/local/dom0 resume=/dev/local/swap pciback.hide=(01:00.0)(02:00.0)(02:00.1) splash=silent quiet showopts [ 3.646529] pciback 0000:01:00.0: seizing device [ 3.646536] pciback 0000:02:00.0: seizing device [ 3.646540] pciback 0000:02:00.1: seizing device [ 3.646594] pciback 0000:02:00.1: PCI INT A -> GSI 16 (level, low) -> IRQ 16 [ 3.646599] pciback 0000:02:00.1: PCI INT A disabled [ 3.646637] pciback 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 [ 3.646642] pciback 0000:02:00.0: PCI INT A disabled So, everything points to the devices having been removed from dom0''s visibility and placed under the control of pciback. A check with "xm" confirms this: # xm pci-list-assignable-devices 0000:02:00.0 0000:02:00.1 0000:01:00.0 Next, I set up the pci pass through in my HVM domU configuration. By the way, VTd is enabled and working fine - xm dmesg shows it as enabled, and, as already mentioned, I can forward through other PCI(e) devices without problem. I set the following in my domU configuration: pci=[''01:00.0'',''02:00.0'',''02:00.1''] and then I try to start the domU: # xm start XP Error: pci: PCI Backend and pci-stub don''t own device 0000:01:00.0 Usage: xm start <DomainName> ??? Ummm, actually, the device is owned by pciback, so why is Xen telling me it''s not?? I can confirm via lspci -v: # lspci -v ... 01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Memory at d0000000 (64-bit, prefetchable) [size=64K] Bus: primary=01, secondary=02, subordinate=02, sec-latency=64 I/O behind bridge: 0000d000-0000dfff Memory behind bridge: fe500000-fe5fffff Capabilities: [40] Power Management version 2 Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [60] Express PCI/PCI-X Bridge, MSI 00 Capabilities: [100] Power Budgeting <?> Kernel driver in use: pciback Any hints on this particular part of the problem would be appreciated. My next step was to leave this one out and just try forwarding through 02:00.0 and 02:00.1. However, when I modify the domU configuration and remove the 01:00.0 device and try to start the domU: # xm start XP Error: Failed to assign device to IOMMU (0000:02:00.0@100,msitranslate=1,power_mgmt=0) Usage: xm start <DomainName> I''ve been able to find very, very little useful information on this particular error - most of the issues are with trying to forward through one or two of the devices on a card to one domU and the others to another or not at all - splitting the devices on a card between domains. I''m not really trying to do this, but I suspect that there''s some sort of tie between the PCIe-PCI bridge device (01:00.0) and the actual serial card (02:00.0) that is causing Xen to need to have both those devices present in the same domain. Of course, because of the problems I''m having forwarding the bridge through, that''s not really working for me. Environment: Xen: 4.0.0 (4.0.0_21091_06-0.1) Kernel: 2.6.34.7 (2.6.34.7-0.4-xen, not pvops) Hardware: Dell Optiplex 780 Any help, hints, information, would be greatly appreciated! -Nick -------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I''m not sure if this will be of help to you. I run a Xenified 2.6.34 kernel and use iommu=pv on dom0 and iommu=soft swiotlb=force on domU and haven''t seen this. You may want to forward lspci -t output, dom0 grub.cfg and domU .cfg to the list and also check the xend-config.sxp file. There is a "restrictive" (don''t recall the exact name) setting that is default for assigning pci devices that I will typically set to no. On 10/21/2010 9:28 AM, Nick Couchman wrote:> I''ve asked a couple of these questions, already, but haven''t really had > any response, so I thought I''d try, again, and try to provide a little > more detail. I need to pass through a PCIe 4-port Serial card. I''ve > done this successfully multiple times with different serial cards, but > the current one, a PCIe Perle quad-port RJ45, is causing some problems. > I believe the problem stems from the fact that the card is a PCI card > and that Perle just built a PCIe-PCI bridge so that they could reuse an > existing PCI design. The following three PCI devices are all on the > same card: > > # lspci > ... > 01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI > Express-to-PCI Bridge (rev aa) > 02:00.0 Serial controller: Perle Systems Ltd Device 0331 > 02:00.1 Bridge: Perle Systems Ltd Device 0331 > > I''ve tried various combinations of hiding 01:00.0, 02:00.0, and/or > 02:00.1 from dom0 and passing those through. First, if I try all three, > I put the following in my dom0 kernel arguments: > > pciback.hide=(01:00.0)(02:00.0)(02:00.1) > > Before someone suggests that maybe I have my kernel arguments wrong, > this is a SuSE kernel, so it is not pvops, and that is the correct > syntax - I can add other PCI(e) devices on the same exact host (like a > graphics card) and pass those through without any problem at all. So > syntax is correct. After I boot with these arguments, I check the dmesg > output for pciback messages: > > # dmesg|grep pciback > [ 0.000000] Command line: root=/dev/local/dom0 resume=/dev/local/swap > pciback.hide=(01:00.0)(02:00.0)(02:00.1) splash=silent quiet showopts > [ 0.000000] Kernel command line: root=/dev/local/dom0 > resume=/dev/local/swap pciback.hide=(01:00.0)(02:00.0)(02:00.1) > splash=silent quiet showopts > [ 3.646529] pciback 0000:01:00.0: seizing device > [ 3.646536] pciback 0000:02:00.0: seizing device > [ 3.646540] pciback 0000:02:00.1: seizing device > [ 3.646594] pciback 0000:02:00.1: PCI INT A -> GSI 16 (level, low) -> > IRQ 16 > [ 3.646599] pciback 0000:02:00.1: PCI INT A disabled > [ 3.646637] pciback 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> > IRQ 16 > [ 3.646642] pciback 0000:02:00.0: PCI INT A disabled > > So, everything points to the devices having been removed from dom0''s > visibility and placed under the control of pciback. A check with "xm" > confirms this: > > # xm pci-list-assignable-devices > 0000:02:00.0 > 0000:02:00.1 > 0000:01:00.0 > > Next, I set up the pci pass through in my HVM domU configuration. By > the way, VTd is enabled and working fine - xm dmesg shows it as enabled, > and, as already mentioned, I can forward through other PCI(e) devices > without problem. I set the following in my domU configuration: > > pci=[''01:00.0'',''02:00.0'',''02:00.1''] > > and then I try to start the domU: > > # xm start XP > Error: pci: PCI Backend and pci-stub don''t own device 0000:01:00.0 > Usage: xm start<DomainName> > > ??? Ummm, actually, the device is owned by pciback, so why is Xen > telling me it''s not?? I can confirm via lspci -v: > > # lspci -v > ... > 01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI > Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Memory at d0000000 (64-bit, prefetchable) [size=64K] > Bus: primary=01, secondary=02, subordinate=02, sec-latency=64 > I/O behind bridge: 0000d000-0000dfff > Memory behind bridge: fe500000-fe5fffff > Capabilities: [40] Power Management version 2 > Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+ > Capabilities: [60] Express PCI/PCI-X Bridge, MSI 00 > Capabilities: [100] Power Budgeting<?> > Kernel driver in use: pciback > > Any hints on this particular part of the problem would be appreciated. > My next step was to leave this one out and just try forwarding through > 02:00.0 and 02:00.1. However, when I modify the domU configuration and > remove the 01:00.0 device and try to start the domU: > > # xm start XP > Error: Failed to assign device to IOMMU > (0000:02:00.0@100,msitranslate=1,power_mgmt=0) > Usage: xm start<DomainName> > > I''ve been able to find very, very little useful information on this > particular error - most of the issues are with trying to forward through > one or two of the devices on a card to one domU and the others to > another or not at all - splitting the devices on a card between domains. > I''m not really trying to do this, but I suspect that there''s some sort > of tie between the PCIe-PCI bridge device (01:00.0) and the actual > serial card (02:00.0) that is causing Xen to need to have both those > devices present in the same domain. Of course, because of the problems > I''m having forwarding the bridge through, that''s not really working for > me. > > Environment: > Xen: 4.0.0 (4.0.0_21091_06-0.1) > Kernel: 2.6.34.7 (2.6.34.7-0.4-xen, not pvops) > Hardware: Dell Optiplex 780 > > Any help, hints, information, would be greatly appreciated! > > -Nick > > > > > -------- > > This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Richie, Thanks for the response - unfortunately it sounds like you''re running a PV domU - I need to forward these PCI devices to an HVM domU, so the iommu=pv option on dom0 and the others for domU won''t really work for me. I''ll check on the restrictive option to see if that will help me - I think I had already played with that option a little bit, but it''s worth another shot! Thanks, Nick On Thu, 2010-10-21 at 09:43 -0400, Richie wrote:> I''m not sure if this will be of help to you. I run a Xenified 2.6.34 > kernel and use iommu=pv on dom0 and iommu=soft swiotlb=force on domU > and haven''t seen this. You may want to forward lspci -t output, dom0 > grub.cfg and domU .cfg to the list and also check the xend-config.sxp > file. There is a "restrictive" (don''t recall the exact name) setting > that is default for assigning pci devices that I will typically set to no. > > On 10/21/2010 9:28 AM, Nick Couchman wrote: > > I''ve asked a couple of these questions, already, but haven''t really had > > any response, so I thought I''d try, again, and try to provide a little > > more detail. I need to pass through a PCIe 4-port Serial card. I''ve > > done this successfully multiple times with different serial cards, but > > the current one, a PCIe Perle quad-port RJ45, is causing some problems. > > I believe the problem stems from the fact that the card is a PCI card > > and that Perle just built a PCIe-PCI bridge so that they could reuse an > > existing PCI design. The following three PCI devices are all on the > > same card: > > > > # lspci > > ... > > 01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI > > Express-to-PCI Bridge (rev aa) > > 02:00.0 Serial controller: Perle Systems Ltd Device 0331 > > 02:00.1 Bridge: Perle Systems Ltd Device 0331 > > > > I''ve tried various combinations of hiding 01:00.0, 02:00.0, and/or > > 02:00.1 from dom0 and passing those through. First, if I try all three, > > I put the following in my dom0 kernel arguments: > > > > pciback.hide=(01:00.0)(02:00.0)(02:00.1) > > > > Before someone suggests that maybe I have my kernel arguments wrong, > > this is a SuSE kernel, so it is not pvops, and that is the correct > > syntax - I can add other PCI(e) devices on the same exact host (like a > > graphics card) and pass those through without any problem at all. So > > syntax is correct. After I boot with these arguments, I check the dmesg > > output for pciback messages: > > > > # dmesg|grep pciback > > [ 0.000000] Command line: root=/dev/local/dom0 resume=/dev/local/swap > > pciback.hide=(01:00.0)(02:00.0)(02:00.1) splash=silent quiet showopts > > [ 0.000000] Kernel command line: root=/dev/local/dom0 > > resume=/dev/local/swap pciback.hide=(01:00.0)(02:00.0)(02:00.1) > > splash=silent quiet showopts > > [ 3.646529] pciback 0000:01:00.0: seizing device > > [ 3.646536] pciback 0000:02:00.0: seizing device > > [ 3.646540] pciback 0000:02:00.1: seizing device > > [ 3.646594] pciback 0000:02:00.1: PCI INT A -> GSI 16 (level, low) -> > > IRQ 16 > > [ 3.646599] pciback 0000:02:00.1: PCI INT A disabled > > [ 3.646637] pciback 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> > > IRQ 16 > > [ 3.646642] pciback 0000:02:00.0: PCI INT A disabled > > > > So, everything points to the devices having been removed from dom0''s > > visibility and placed under the control of pciback. A check with "xm" > > confirms this: > > > > # xm pci-list-assignable-devices > > 0000:02:00.0 > > 0000:02:00.1 > > 0000:01:00.0 > > > > Next, I set up the pci pass through in my HVM domU configuration. By > > the way, VTd is enabled and working fine - xm dmesg shows it as enabled, > > and, as already mentioned, I can forward through other PCI(e) devices > > without problem. I set the following in my domU configuration: > > > > pci=[''01:00.0'',''02:00.0'',''02:00.1''] > > > > and then I try to start the domU: > > > > # xm start XP > > Error: pci: PCI Backend and pci-stub don''t own device 0000:01:00.0 > > Usage: xm start<DomainName> > > > > ??? Ummm, actually, the device is owned by pciback, so why is Xen > > telling me it''s not?? I can confirm via lspci -v: > > > > # lspci -v > > ... > > 01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI > > Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode]) > > Flags: bus master, fast devsel, latency 0 > > Memory at d0000000 (64-bit, prefetchable) [size=64K] > > Bus: primary=01, secondary=02, subordinate=02, sec-latency=64 > > I/O behind bridge: 0000d000-0000dfff > > Memory behind bridge: fe500000-fe5fffff > > Capabilities: [40] Power Management version 2 > > Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+ > > Capabilities: [60] Express PCI/PCI-X Bridge, MSI 00 > > Capabilities: [100] Power Budgeting<?> > > Kernel driver in use: pciback > > > > Any hints on this particular part of the problem would be appreciated. > > My next step was to leave this one out and just try forwarding through > > 02:00.0 and 02:00.1. However, when I modify the domU configuration and > > remove the 01:00.0 device and try to start the domU: > > > > # xm start XP > > Error: Failed to assign device to IOMMU > > (0000:02:00.0@100,msitranslate=1,power_mgmt=0) > > Usage: xm start<DomainName> > > > > I''ve been able to find very, very little useful information on this > > particular error - most of the issues are with trying to forward through > > one or two of the devices on a card to one domU and the others to > > another or not at all - splitting the devices on a card between domains. > > I''m not really trying to do this, but I suspect that there''s some sort > > of tie between the PCIe-PCI bridge device (01:00.0) and the actual > > serial card (02:00.0) that is causing Xen to need to have both those > > devices present in the same domain. Of course, because of the problems > > I''m having forwarding the bridge through, that''s not really working for > > me. > > > > Environment: > > Xen: 4.0.0 (4.0.0_21091_06-0.1) > > Kernel: 2.6.34.7 (2.6.34.7-0.4-xen, not pvops) > > Hardware: Dell Optiplex 780 > > > > Any help, hints, information, would be greatly appreciated! > > > > -Nick > > > > > > > > > > -------- > > > > This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users >-------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 10/21/2010 9:45 AM, Nick Couchman wrote:> Richie, > Thanks for the response - unfortunately it sounds like you''re running a > PV domU - I need to forward these PCI devices to an HVM domU, so the > iommu=pv option on dom0 and the others for domU won''t really work for > me. I''ll check on the restrictive option to see if that will help me - > I think I had already played with that option a little bit, but it''s > worth another shot! > > Thanks, > Nick >I suppose the msitranslate=1 could also play a part in this, but I don''t use that option nor know much of anything about it. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, 2010-10-21 at 10:00 -0400, Richie wrote:> On 10/21/2010 9:45 AM, Nick Couchman wrote: > > Richie, > > Thanks for the response - unfortunately it sounds like you''re running a > > PV domU - I need to forward these PCI devices to an HVM domU, so the > > iommu=pv option on dom0 and the others for domU won''t really work for > > me. I''ll check on the restrictive option to see if that will help me - > > I think I had already played with that option a little bit, but it''s > > worth another shot! > > > > Thanks, > > Nick > > > I suppose the msitranslate=1 could also play a part in this, but I don''t > use that option nor know much of anything about it. > >I have played with msitranslate, and setting different values for that has not helped. -Nick -------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users