This is a message I posted to xen-users without getting any good leads...I''m hoping someone on the devel list has some ideas as to what might be going on here. I''ve asked a couple of these questions, already, but haven''t really had any response, so I thought I''d try, again, and try to provide a little more detail. I need to pass through a PCIe 4-port Serial card. I''ve done this successfully multiple times with different serial cards, but the current one, a PCIe Perle quad-port RJ45, is causing some problems. I believe the problem stems from the fact that the card is a PCI card and that Perle just built a PCIe-PCI bridge so that they could reuse an existing PCI design. The following three PCI devices are all on the same card: # lspci ... 01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge (rev aa) 02:00.0 Serial controller: Perle Systems Ltd Device 0331 02:00.1 Bridge: Perle Systems Ltd Device 0331 I''ve tried various combinations of hiding 01:00.0, 02:00.0, and/or 02:00.1 from dom0 and passing those through. First, if I try all three, I put the following in my dom0 kernel arguments: pciback.hide=(01:00.0)(02:00.0)(02:00.1) Before someone suggests that maybe I have my kernel arguments wrong, this is a SuSE kernel, so it is not pvops, and that is the correct syntax - I can add other PCI(e) devices on the same exact host (like a graphics card) and pass those through without any problem at all. So syntax is correct. After I boot with these arguments, I check the dmesg output for pciback messages: # dmesg|grep pciback [ 0.000000] Command line: root=/dev/local/dom0 resume=/dev/local/swap pciback.hide=(01:00.0)(02:00.0)(02:00.1) splash=silent quiet showopts [ 0.000000] Kernel command line: root=/dev/local/dom0 resume=/dev/local/swap pciback.hide=(01:00.0)(02:00.0)(02:00.1) splash=silent quiet showopts [ 3.646529] pciback 0000:01:00.0: seizing device [ 3.646536] pciback 0000:02:00.0: seizing device [ 3.646540] pciback 0000:02:00.1: seizing device [ 3.646594] pciback 0000:02:00.1: PCI INT A -> GSI 16 (level, low) -> IRQ 16 [ 3.646599] pciback 0000:02:00.1: PCI INT A disabled [ 3.646637] pciback 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 [ 3.646642] pciback 0000:02:00.0: PCI INT A disabled So, everything points to the devices having been removed from dom0''s visibility and placed under the control of pciback. A check with "xm" confirms this: # xm pci-list-assignable-devices 0000:02:00.0 0000:02:00.1 0000:01:00.0 Next, I set up the pci pass through in my HVM domU configuration. By the way, VTd is enabled and working fine - xm dmesg shows it as enabled, and, as already mentioned, I can forward through other PCI(e) devices without problem. I set the following in my domU configuration: pci=[''01:00.0'',''02:00.0'',''02:00.1''] and then I try to start the domU: # xm start XP Error: pci: PCI Backend and pci-stub don''t own device 0000:01:00.0 Usage: xm start <DomainName> ??? Ummm, actually, the device is owned by pciback, so why is Xen telling me it''s not?? I can confirm via lspci -v: # lspci -v ... 01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Memory at d0000000 (64-bit, prefetchable) [size=64K] Bus: primary=01, secondary=02, subordinate=02, sec-latency=64 I/O behind bridge: 0000d000-0000dfff Memory behind bridge: fe500000-fe5fffff Capabilities: [40] Power Management version 2 Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [60] Express PCI/PCI-X Bridge, MSI 00 Capabilities: [100] Power Budgeting <?> Kernel driver in use: pciback Any hints on this particular part of the problem would be appreciated. My next step was to leave this one out and just try forwarding through 02:00.0 and 02:00.1. However, when I modify the domU configuration and remove the 01:00.0 device and try to start the domU: # xm start XP Error: Failed to assign device to IOMMU (0000:02:00.0@100,msitranslate=1,power_mgmt=0) Usage: xm start <DomainName> I''ve been able to find very, very little useful information on this particular error - most of the issues are with trying to forward through one or two of the devices on a card to one domU and the others to another or not at all - splitting the devices on a card between domains. I''m not really trying to do this, but I suspect that there''s some sort of tie between the PCIe-PCI bridge device (01:00.0) and the actual serial card (02:00.0) that is causing Xen to need to have both those devices present in the same domain. Of course, because of the problems I''m having forwarding the bridge through, that''s not really working for me. Environment: Xen: 4.0.0 (4.0.0_21091_06-0.1) Kernel: 2.6.34.7 (2.6.34.7-0.4-xen, not pvops) Hardware: Dell Optiplex 780 Any help, hints, information, would be greatly appreciated! -Nick -------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Oct-25 17:40 UTC
Re: [Xen-devel] PCI Passthrough Problems/Questions
On Mon, Oct 25, 2010 at 10:35:08AM -0600, Nick Couchman wrote:> This is a message I posted to xen-users without getting any good leads...I''m hoping someone on the devel list has some ideas as to what might be going on here. > > I''ve asked a couple of these questions, already, but haven''t really had > any response, so I thought I''d try, again, and try to provide a little > more detail. I need to pass through a PCIe 4-port Serial card. I''ve > done this successfully multiple times with different serial cards, but > the current one, a PCIe Perle quad-port RJ45, is causing some problems. > I believe the problem stems from the fact that the card is a PCI card > and that Perle just built a PCIe-PCI bridge so that they could reuse an > existing PCI design. The following three PCI devices are all on the > same card: > > # lspci > ... > 01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI > Express-to-PCI Bridge (rev aa) > 02:00.0 Serial controller: Perle Systems Ltd Device 0331 > 02:00.1 Bridge: Perle Systems Ltd Device 0331 > > I''ve tried various combinations of hiding 01:00.0, 02:00.0, and/or > 02:00.1 from dom0 and passing those through. First, if I try all three, > I put the following in my dom0 kernel arguments: > > pciback.hide=(01:00.0)(02:00.0)(02:00.1) > > Before someone suggests that maybe I have my kernel arguments wrong, > this is a SuSE kernel, so it is not pvops, and that is the correct > syntax - I can add other PCI(e) devices on the same exact host (like a > graphics card) and pass those through without any problem at all. So > syntax is correct. After I boot with these arguments, I check the dmesg > output for pciback messages: > > # dmesg|grep pciback > [ 0.000000] Command line: root=/dev/local/dom0 resume=/dev/local/swap > pciback.hide=(01:00.0)(02:00.0)(02:00.1) splash=silent quiet showopts > [ 0.000000] Kernel command line: root=/dev/local/dom0 > resume=/dev/local/swap pciback.hide=(01:00.0)(02:00.0)(02:00.1) > splash=silent quiet showopts > [ 3.646529] pciback 0000:01:00.0: seizing device > [ 3.646536] pciback 0000:02:00.0: seizing device > [ 3.646540] pciback 0000:02:00.1: seizing device > [ 3.646594] pciback 0000:02:00.1: PCI INT A -> GSI 16 (level, low) -> > IRQ 16 > [ 3.646599] pciback 0000:02:00.1: PCI INT A disabled > [ 3.646637] pciback 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> > IRQ 16 > [ 3.646642] pciback 0000:02:00.0: PCI INT A disabled > > So, everything points to the devices having been removed from dom0''s > visibility and placed under the control of pciback. A check with "xm" > confirms this: > > # xm pci-list-assignable-devices > 0000:02:00.0 > 0000:02:00.1 > 0000:01:00.0 > > Next, I set up the pci pass through in my HVM domU configuration. By > the way, VTd is enabled and working fine - xm dmesg shows it as enabled, > and, as already mentioned, I can forward through other PCI(e) devices > without problem. I set the following in my domU configuration: > > pci=[''01:00.0'',''02:00.0'',''02:00.1''] > > and then I try to start the domU: > > # xm start XP > Error: pci: PCI Backend and pci-stub don''t own device 0000:01:00.0 > Usage: xm start <DomainName> > > ??? Ummm, actually, the device is owned by pciback, so why is Xen > telling me it''s not?? I can confirm via lspci -v: > > # lspci -v > ... > 01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI > Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode]) > Flags: bus master, fast devsel, latency 0 > Memory at d0000000 (64-bit, prefetchable) [size=64K] > Bus: primary=01, secondary=02, subordinate=02, sec-latency=64 > I/O behind bridge: 0000d000-0000dfff > Memory behind bridge: fe500000-fe5fffff > Capabilities: [40] Power Management version 2 > Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+ > Capabilities: [60] Express PCI/PCI-X Bridge, MSI 00 > Capabilities: [100] Power Budgeting <?> > Kernel driver in use: pciback > > Any hints on this particular part of the problem would be appreciated. > My next step was to leave this one out and just try forwarding through > 02:00.0 and 02:00.1. However, when I modify the domU configuration and > remove the 01:00.0 device and try to start the domU: > > # xm start XP > Error: Failed to assign device to IOMMU > (0000:02:00.0@100,msitranslate=1,power_mgmt=0) > Usage: xm start <DomainName> > > I''ve been able to find very, very little useful information on this > particular error - most of the issues are with trying to forward through > one or two of the devices on a card to one domU and the others to > another or not at all - splitting the devices on a card between domains. > I''m not really trying to do this, but I suspect that there''s some sort > of tie between the PCIe-PCI bridge device (01:00.0) and the actual > serial card (02:00.0) that is causing Xen to need to have both those > devices present in the same domain. Of course, because of the problems > I''m having forwarding the bridge through, that''s not really working for > me.What do you see on your Xen serial output? I presume you cranked up logging: loglevel=all guest_lvl=all iommu=verbose on your Xen command line. Is there anything that shows up when you get the ''Failed to assign.." ? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, 2010-10-25 at 13:40 -0400, Konrad Rzeszutek Wilk wrote:> > What do you see on your Xen serial output? I presume you cranked up logging: > loglevel=all guest_lvl=all iommu=verbose on your Xen command line. > > Is there anything that shows up when you get the ''Failed to assign.." ? >The only messages I get on the serial console after setting those parameters on the xen.gz kernel line in grub (and rebooting, of course) are the following: (XEN) [VT-D]iommu.c:1496: d0:PCI: unmap bdf = 2:0.0 (XEN) [VT-D]iommu.c:1364: d1:PCI: map bdf = 2:0.0 (XEN) domctl.c:848:d0 XEN_DOMCTL_assign_device: assign device (2:0.0) failed (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 1, error -22 -Nick -------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Oct-25 18:48 UTC
Re: [Xen-devel] PCI Passthrough Problems/Questions
On Mon, Oct 25, 2010 at 12:33:09PM -0600, Nick Couchman wrote:> On Mon, 2010-10-25 at 13:40 -0400, Konrad Rzeszutek Wilk wrote: > > > > > What do you see on your Xen serial output? I presume you cranked up logging: > > loglevel=all guest_lvl=all iommu=verbose on your Xen command line. > > > > Is there anything that shows up when you get the ''Failed to assign.." ? > > > > The only messages I get on the serial console after setting those > parameters on the xen.gz kernel line in grub (and rebooting, of course) > are the following: > > (XEN) [VT-D]iommu.c:1496: d0:PCI: unmap bdf = 2:0.0 > (XEN) [VT-D]iommu.c:1364: d1:PCI: map bdf = 2:0.0 > (XEN) domctl.c:848:d0 XEN_DOMCTL_assign_device: assign device (2:0.0) > failed > (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 1, error -22So, -EINVAL. How comfortable are you sticking a bunch of dprintk(VTDPREFIX, " in the drivers/passthrough/vtd/iommu.c file? Basically you need to figure which of the functions that are past line 1364 are being called and return -EINVAL. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, 2010-10-25 at 14:48 -0400, Konrad Rzeszutek Wilk wrote:> On Mon, Oct 25, 2010 at 12:33:09PM -0600, Nick Couchman wrote: > > On Mon, 2010-10-25 at 13:40 -0400, Konrad Rzeszutek Wilk wrote: > > > > > > > > What do you see on your Xen serial output? I presume you cranked up logging: > > > loglevel=all guest_lvl=all iommu=verbose on your Xen command line. > > > > > > Is there anything that shows up when you get the ''Failed to assign.." ? > > > > > > > The only messages I get on the serial console after setting those > > parameters on the xen.gz kernel line in grub (and rebooting, of course) > > are the following: > > > > (XEN) [VT-D]iommu.c:1496: d0:PCI: unmap bdf = 2:0.0 > > (XEN) [VT-D]iommu.c:1364: d1:PCI: map bdf = 2:0.0 > > (XEN) domctl.c:848:d0 XEN_DOMCTL_assign_device: assign device (2:0.0) > > failed > > (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 1, error -22 > > So, -EINVAL. How comfortable are you sticking a bunch of > dprintk(VTDPREFIX, " in the drivers/passthrough/vtd/iommu.c file? Basically > you need to figure which of the functions that are past line 1364 > are being called and return -EINVAL.I''m happy to give it a shot...it''ll take a while to get the devel environment configured, as I''m using packages right now and I don''t even think I have a compiler on this system. I''ll report back once I get that done and give that a try. -Nick -------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Oct-25 19:07 UTC
Re: [Xen-devel] PCI Passthrough Problems/Questions
On Mon, Oct 25, 2010 at 12:50:41PM -0600, Nick Couchman wrote:> On Mon, 2010-10-25 at 14:48 -0400, Konrad Rzeszutek Wilk wrote: > > On Mon, Oct 25, 2010 at 12:33:09PM -0600, Nick Couchman wrote: > > > On Mon, 2010-10-25 at 13:40 -0400, Konrad Rzeszutek Wilk wrote: > > > > > > > > > > > What do you see on your Xen serial output? I presume you cranked up logging: > > > > loglevel=all guest_lvl=all iommu=verbose on your Xen command line. > > > > > > > > Is there anything that shows up when you get the ''Failed to assign.." ? > > > > > > > > > > The only messages I get on the serial console after setting those > > > parameters on the xen.gz kernel line in grub (and rebooting, of course) > > > are the following: > > > > > > (XEN) [VT-D]iommu.c:1496: d0:PCI: unmap bdf = 2:0.0 > > > (XEN) [VT-D]iommu.c:1364: d1:PCI: map bdf = 2:0.0 > > > (XEN) domctl.c:848:d0 XEN_DOMCTL_assign_device: assign device (2:0.0) > > > failed > > > (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 1, error -22 > > > > So, -EINVAL. How comfortable are you sticking a bunch of > > dprintk(VTDPREFIX, " in the drivers/passthrough/vtd/iommu.c file? Basically > > you need to figure which of the functions that are past line 1364 > > are being called and return -EINVAL. > > I''m happy to give it a shot...it''ll take a while to get the devel > environment configured, as I''m using packages right now and I don''t even > think I have a compiler on this system. I''ll report back once I get > that done and give that a try.Excellent. You might also want to CC Weidong (weidong.han@intel.com) in the future who is right now on travel and he might have better suggestions. CC-ing him on this e-mail. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, 2010-10-25 at 15:07 -0400, Konrad Rzeszutek Wilk wrote:> On Mon, Oct 25, 2010 at 12:50:41PM -0600, Nick Couchman wrote: > > On Mon, 2010-10-25 at 14:48 -0400, Konrad Rzeszutek Wilk wrote: > > > On Mon, Oct 25, 2010 at 12:33:09PM -0600, Nick Couchman wrote: > > > > On Mon, 2010-10-25 at 13:40 -0400, Konrad Rzeszutek Wilk wrote: > > > > > > > > > > > > > > What do you see on your Xen serial output? I presume you cranked up logging: > > > > > loglevel=all guest_lvl=all iommu=verbose on your Xen command line. > > > > > > > > > > Is there anything that shows up when you get the ''Failed to assign.." ? > > > > > > > > > > > > > The only messages I get on the serial console after setting those > > > > parameters on the xen.gz kernel line in grub (and rebooting, of course) > > > > are the following: > > > > > > > > (XEN) [VT-D]iommu.c:1496: d0:PCI: unmap bdf = 2:0.0 > > > > (XEN) [VT-D]iommu.c:1364: d1:PCI: map bdf = 2:0.0 > > > > (XEN) domctl.c:848:d0 XEN_DOMCTL_assign_device: assign device (2:0.0) > > > > failed > > > > (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 1, error -22 > > > > > > So, -EINVAL. How comfortable are you sticking a bunch of > > > dprintk(VTDPREFIX, " in the drivers/passthrough/vtd/iommu.c file? Basically > > > you need to figure which of the functions that are past line 1364 > > > are being called and return -EINVAL. > > > > I''m happy to give it a shot...it''ll take a while to get the devel > > environment configured, as I''m using packages right now and I don''t even > > think I have a compiler on this system. I''ll report back once I get > > that done and give that a try. > > Excellent. You might also want to CC Weidong (weidong.han@intel.com) in the future > who is right now on travel and he might have better suggestions. CC-ing him on this e-mail.Okay, I''ve traced it down to this section of code in iommu.c: /* * Devices behind PCIe-to-PCI/PCIx bridge may generate * different requester-id. It may originate from devfn=0 * on the secondary bus behind the bridge. Map that id * as well. */ ret = domain_context_mapping_one(domain, drhd->iommu, secbus, 0); dprintk(XENLOG_ERR VTDPREFIX, "domain_conext_mapping_one ret: %d\n", ret); I guess that shouldn''t surprise me, given this device is behind a PCIe-to-PCI bridge. On to the domain_context_mapping_one function. -Nick -------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, 2010-10-25 at 15:07 -0400, Konrad Rzeszutek Wilk wrote:> On Mon, Oct 25, 2010 at 12:50:41PM -0600, Nick Couchman wrote: > > On Mon, 2010-10-25 at 14:48 -0400, Konrad Rzeszutek Wilk wrote: > > > On Mon, Oct 25, 2010 at 12:33:09PM -0600, Nick Couchman wrote: > > > > On Mon, 2010-10-25 at 13:40 -0400, Konrad Rzeszutek Wilk wrote: > > > > > > > > > > > > > > What do you see on your Xen serial output? I presume you cranked up logging: > > > > > loglevel=all guest_lvl=all iommu=verbose on your Xen command line. > > > > > > > > > > Is there anything that shows up when you get the ''Failed to assign.." ? > > > > > > > > > > > > > The only messages I get on the serial console after setting those > > > > parameters on the xen.gz kernel line in grub (and rebooting, of course) > > > > are the following: > > > > > > > > (XEN) [VT-D]iommu.c:1496: d0:PCI: unmap bdf = 2:0.0 > > > > (XEN) [VT-D]iommu.c:1364: d1:PCI: map bdf = 2:0.0 > > > > (XEN) domctl.c:848:d0 XEN_DOMCTL_assign_device: assign device (2:0.0) > > > > failed > > > > (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 1, error -22 > > > > > > So, -EINVAL. How comfortable are you sticking a bunch of > > > dprintk(VTDPREFIX, " in the drivers/passthrough/vtd/iommu.c file? Basically > > > you need to figure which of the functions that are past line 1364 > > > are being called and return -EINVAL. > > > > I''m happy to give it a shot...it''ll take a while to get the devel > > environment configured, as I''m using packages right now and I don''t even > > think I have a compiler on this system. I''ll report back once I get > > that done and give that a try. > > Excellent. You might also want to CC Weidong (weidong.han@intel.com) in the future > who is right now on travel and he might have better suggestions. CC-ing him on this e-mail.Okay, I''m fairly certain that the root cause is actually the issue Xen has trying to forward through the actual PCIe-to-PCI bridge. In my original e-mail, I stated that if I try to add that bridge to the domU, I get an error about it not being owned by PCI Backend or pci-stub. In the function domain_context_mapping, bus is the bus of the device you''re actually trying to forward through to the new domain, and secbus is set with the function find_upstream_bridge, which appears to find the bus to which your device is attached. Here''s the section of code in question from iommu.c: if ( pdev_type(bus, devfn) == DEV_TYPE_PCIe2PCI_BRIDGE ) { ret = domain_context_mapping_one(domain, drhd->iommu, bus, devfn); dprintk(XENLOG_ERR VTDPREFIX, "domain_conext_mapping_one ret: %d\n", ret); if ( ret ) return ret; /* * Devices behind PCIe-to-PCI/PCIx bridge may generate * different requester-id. It may originate from devfn=0 * on the secondary bus behind the bridge. Map that id * as well. */ ret = domain_context_mapping_one(domain, drhd->iommu, secbus, 0); dprintk(XENLOG_ERR VTDPREFIX, "domain_conext_mapping_one ret: %d\n", ret); } else /* Legacy PCI bridge */ { ret = domain_context_mapping_one(domain, drhd->iommu, bus, devfn); dprintk(XENLOG_ERR VTDPREFIX, "domain_conext_mapping_one ret: %d\n", ret); } If I understand this code, if the device is a PCIe2PCI bridge, the first thing that''s done is that the device itself is mapped, then the upstream bridge device is mapped (secbus with devfn=0). This is where the command is failing for the device that I''m trying to forward. So, the question is, why can''t Xen seem to forward through the bridge itself? It doesn''t seem to matter whether I have that particular bridge listed on the kernel line in the pciback.hide section - Xen still thinks that dom0 has a hold of that device and won''t forward it on. Where do I go from here? -Nick -------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk wrote:> On Mon, Oct 25, 2010 at 12:50:41PM -0600, Nick Couchman wrote: > >> On Mon, 2010-10-25 at 14:48 -0400, Konrad Rzeszutek Wilk wrote: >> >>> On Mon, Oct 25, 2010 at 12:33:09PM -0600, Nick Couchman wrote: >>> >>>> On Mon, 2010-10-25 at 13:40 -0400, Konrad Rzeszutek Wilk wrote: >>>> >>>> >>>>> What do you see on your Xen serial output? I presume you cranked up logging: >>>>> loglevel=all guest_lvl=all iommu=verbose on your Xen command line. >>>>> >>>>> Is there anything that shows up when you get the ''Failed to assign.." ? >>>>> >>>>> >>>> The only messages I get on the serial console after setting those >>>> parameters on the xen.gz kernel line in grub (and rebooting, of course) >>>> are the following: >>>> >>>> (XEN) [VT-D]iommu.c:1496: d0:PCI: unmap bdf = 2:0.0 >>>> (XEN) [VT-D]iommu.c:1364: d1:PCI: map bdf = 2:0.0 >>>> (XEN) domctl.c:848:d0 XEN_DOMCTL_assign_device: assign device (2:0.0) >>>> failed >>>> (XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 1, error -22 >>>>Nick, I think the issue is 02:00.0 was mapped twice. Could you try with below patch? Then post the xen log. Pls post all output of ''lspci -v'' on your system. diff -r eff592364826 xen/drivers/passthrough/vtd/iommu.c --- a/xen/drivers/passthrough/vtd/iommu.c Wed Sep 01 11:23:49 2010 +0100 +++ b/xen/drivers/passthrough/vtd/iommu.c Wed Oct 27 14:33:23 2010 +0800 @@ -1225,7 +1225,11 @@ static int domain_context_mapping_one( if (!pdev) res = -ENODEV; else if (pdev->domain != domain) - res = -EINVAL; + { + dprintk(VTDPREFIX, "context_present: %x:%x.%x: pdev->domain=%d domain=%d\n", + bus, PCI_SLOT(devfn), PCI_FUNC(devfn), pdev->domain, domain->domain_id); + // res = -EINVAL; + } unmap_vtd_domain_page(context_entries); spin_unlock(&iommu->lock); return res; @@ -1352,6 +1356,8 @@ static int domain_context_mapping(struct /* PCIe to PCI/PCIx bridge */ if ( pdev_type(bus, devfn) == DEV_TYPE_PCIe2PCI_BRIDGE ) { + dprintk(VTDPREFIX, "d%d:PCI: map PCIe2PCI bdf = %x:%x.%x\n", + domain->domain_id, bus, PCI_SLOT(devfn), PCI_FUNC(devfn)); ret = domain_context_mapping_one(domain, drhd->iommu, bus, devfn); if ( ret ) return ret; @@ -1362,6 +1368,8 @@ static int domain_context_mapping(struct * on the secondary bus behind the bridge. Map that id * as well. */ + dprintk(VTDPREFIX, "d%d:PCI: map secbus (%d) with devfn 0\n", + domain->domain_id, secbus); ret = domain_context_mapping_one(domain, drhd->iommu, secbus, 0); } else /* Legacy PCI bridge */>>> So, -EINVAL. How comfortable are you sticking a bunch of >>> dprintk(VTDPREFIX, " in the drivers/passthrough/vtd/iommu.c file? Basically >>> you need to figure which of the functions that are past line 1364 >>> are being called and return -EINVAL. >>> >> I''m happy to give it a shot...it''ll take a while to get the devel >> environment configured, as I''m using packages right now and I don''t even >> think I have a compiler on this system. I''ll report back once I get >> that done and give that a try. >> > > Excellent. You might also want to CC Weidong (weidong.han@intel.com) in the future > who is right now on travel and he might have better suggestions. CC-ing him on this e-mail. >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Nick, > > I think the issue is 02:00.0 was mapped twice. Could you try with below > patch? Then post the xen log. Pls post all output of ''lspci -v'' on your > system. >I applied the patch, with one minor change. The line: dprintk(XENLOG_ERR VTDPREFIX, "context_present: %x:%x.%x:pdev->domain=%d domain=%d\n", bus, PCI_SLOT(devfn), PCI_FUNC(devfn), pdev->domain, domain->domain_id); should be: dprintk(XENLOG_ERR VTDPREFIX, "context_present: %x:%x.%x:pdev->domain=%d domain=%d\n", bus, PCI_SLOT(devfn), PCI_FUNC(devfn), pdev->domain->domain_id, domain->domain_id); (notice the pdev->domain->domain_id instead of pdev->domain). The domU no longer generates the error about failing to assign device to IOMMU, but now it just silently crashes. xm dmesg output: (XEN) [VT-D]iommu.c:1511: d0:PCI: unmap bdf = 2:0.0 (XEN) [VT-D]iommu.c:1340: bus: 2, devfn: 0[VT-D]iommu.c:1368: d1:PCI: map bdf = 2:0.0 (XEN) [VT-D]iommu.c:1371: domain_conext_mapping_one ret: 0 (XEN) [VT-D]iommu.c:1378: Upstream bridge for 1:0 is 2. (XEN) [VT-D]iommu.c:1383: d1:PCI: map PCIe2PCI bdf = 1:0.0 (XEN) [VT-D]iommu.c:1394: d1:PCI: map secbus (2) with devfn 0 (XEN) [VT-D]iommu.c:1249: context_present: 2:0.0:pdev->domain=0 domain=1 (XEN) [VT-D]iommu.c:1415: Return value: 0 (XEN) [VT-D]io.c:300: d1: bind: m_gsi=16 g_gsi=36 device=5 intx=0 (XEN) [VT-D]iommu.c:1511: d0:PCI: unmap bdf = 2:0.1 (XEN) [VT-D]iommu.c:1340: bus: 2, devfn: 1[VT-D]iommu.c:1368: d1:PCI: map bdf = 2:0.1 (XEN) [VT-D]iommu.c:1371: domain_conext_mapping_one ret: 0 (XEN) [VT-D]iommu.c:1378: Upstream bridge for 1:0 is 2. (XEN) [VT-D]iommu.c:1383: d1:PCI: map PCIe2PCI bdf = 1:0.0 (XEN) [VT-D]iommu.c:1394: d1:PCI: map secbus (2) with devfn 0 (XEN) [VT-D]iommu.c:1415: Return value: 0 (XEN) [VT-D]io.c:300: d1: bind: m_gsi=16 g_gsi=40 device=6 intx=0 (XEN) [VT-D]iommu.c:1511: d1:PCI: unmap bdf = 2:0.1 (XEN) [VT-D]iommu.c:1340: bus: 2, devfn: 1[VT-D]iommu.c:1368: d0:PCI: map bdf = 2:0.1 (XEN) [VT-D]iommu.c:1371: domain_conext_mapping_one ret: 0 (XEN) [VT-D]iommu.c:1378: Upstream bridge for 1:0 is 2. (XEN) [VT-D]iommu.c:1383: d0:PCI: map PCIe2PCI bdf = 1:0.0 (XEN) [VT-D]iommu.c:1394: d0:PCI: map secbus (2) with devfn 0 (XEN) [VT-D]iommu.c:1415: Return value: 0 (XEN) [VT-D]iommu.c:1511: d1:PCI: unmap bdf = 2:0.0 (XEN) [VT-D]iommu.c:1340: bus: 2, devfn: 0[VT-D]iommu.c:1368: d0:PCI: map bdf = 2:0.0 (XEN) [VT-D]iommu.c:1371: domain_conext_mapping_one ret: 0 (XEN) [VT-D]iommu.c:1378: Upstream bridge for 1:0 is 2. (XEN) [VT-D]iommu.c:1383: d0:PCI: map PCIe2PCI bdf = 1:0.0 (XEN) [VT-D]iommu.c:1394: d0:PCI: map secbus (2) with devfn 0 (XEN) [VT-D]iommu.c:1249: context_present: 2:0.0:pdev->domain=1 domain=0 (XEN) [VT-D]iommu.c:1415: Return value: 0 And from xend.log I''ve pasted into this pastebin: http://pastebin.com/b4bwdBPq I have some extra dprintk calls that I threw in there, so there may be a little more output than with the patch you sent. -Nick -------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Nick Couchman wrote:>> Nick, >> >> I think the issue is 02:00.0 was mapped twice. Could you try with below >> patch? Then post the xen log. Pls post all output of ''lspci -v'' on your >> system. >> >> > > I applied the patch, with one minor change. The line: > > dprintk(XENLOG_ERR VTDPREFIX, "context_present: %x:%x.%x:pdev->domain=%d > domain=%d\n", bus, PCI_SLOT(devfn), PCI_FUNC(devfn), pdev->domain, > domain->domain_id); > > should be: > > dprintk(XENLOG_ERR VTDPREFIX, "context_present: %x:%x.%x:pdev->domain=%d > domain=%d\n", bus, PCI_SLOT(devfn), PCI_FUNC(devfn), > pdev->domain->domain_id, domain->domain_id); > > (notice the pdev->domain->domain_id instead of pdev->domain). > > The domU no longer generates the error about failing to assign device to > IOMMU, but now it just silently crashes. xm dmesg output: > > (XEN) [VT-D]iommu.c:1511: d0:PCI: unmap bdf = 2:0.0 > (XEN) [VT-D]iommu.c:1340: bus: 2, devfn: 0[VT-D]iommu.c:1368: d1:PCI: > map bdf = 2:0.0 > (XEN) [VT-D]iommu.c:1371: domain_conext_mapping_one ret: 0 > (XEN) [VT-D]iommu.c:1378: Upstream bridge for 1:0 is 2. > (XEN) [VT-D]iommu.c:1383: d1:PCI: map PCIe2PCI bdf = 1:0.0 > (XEN) [VT-D]iommu.c:1394: d1:PCI: map secbus (2) with devfn 0 > (XEN) [VT-D]iommu.c:1249: context_present: 2:0.0:pdev->domain=0 domain=1 > (XEN) [VT-D]iommu.c:1415: Return value: 0 > (XEN) [VT-D]io.c:300: d1: bind: m_gsi=16 g_gsi=36 device=5 intx=0 > (XEN) [VT-D]iommu.c:1511: d0:PCI: unmap bdf = 2:0.1 > (XEN) [VT-D]iommu.c:1340: bus: 2, devfn: 1[VT-D]iommu.c:1368: d1:PCI: > map bdf = 2:0.1 > (XEN) [VT-D]iommu.c:1371: domain_conext_mapping_one ret: 0 > (XEN) [VT-D]iommu.c:1378: Upstream bridge for 1:0 is 2. > (XEN) [VT-D]iommu.c:1383: d1:PCI: map PCIe2PCI bdf = 1:0.0 > (XEN) [VT-D]iommu.c:1394: d1:PCI: map secbus (2) with devfn 0 > (XEN) [VT-D]iommu.c:1415: Return value: 0 > (XEN) [VT-D]io.c:300: d1: bind: m_gsi=16 g_gsi=40 device=6 intx=0 > (XEN) [VT-D]iommu.c:1511: d1:PCI: unmap bdf = 2:0.1 > (XEN) [VT-D]iommu.c:1340: bus: 2, devfn: 1[VT-D]iommu.c:1368: d0:PCI: > map bdf = 2:0.1 > (XEN) [VT-D]iommu.c:1371: domain_conext_mapping_one ret: 0 > (XEN) [VT-D]iommu.c:1378: Upstream bridge for 1:0 is 2. > (XEN) [VT-D]iommu.c:1383: d0:PCI: map PCIe2PCI bdf = 1:0.0 > (XEN) [VT-D]iommu.c:1394: d0:PCI: map secbus (2) with devfn 0 > (XEN) [VT-D]iommu.c:1415: Return value: 0 > (XEN) [VT-D]iommu.c:1511: d1:PCI: unmap bdf = 2:0.0 > (XEN) [VT-D]iommu.c:1340: bus: 2, devfn: 0[VT-D]iommu.c:1368: d0:PCI: > map bdf = 2:0.0 > (XEN) [VT-D]iommu.c:1371: domain_conext_mapping_one ret: 0 > (XEN) [VT-D]iommu.c:1378: Upstream bridge for 1:0 is 2. > (XEN) [VT-D]iommu.c:1383: d0:PCI: map PCIe2PCI bdf = 1:0.0 > (XEN) [VT-D]iommu.c:1394: d0:PCI: map secbus (2) with devfn 0 > (XEN) [VT-D]iommu.c:1249: context_present: 2:0.0:pdev->domain=1 domain=0 > (XEN) [VT-D]iommu.c:1415: Return value: 0 > > And from xend.log I''ve pasted into this pastebin: > > http://pastebin.com/b4bwdBPq > > I have some extra dprintk calls that I threw in there, so there may be a > little more output than with the patch you sent. > > >the log of xm dmesg looks no problem. you''d better post complete log and lspci -v. Regards, Weidong _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I don''t want xend.log. I want xen log (xm dmesg or captured by serial port). And pls also post qemu log (/var/log/xen/dm-qemu-xxx.log). Regards, Weidong -----Original Message----- From: Nick Couchman [mailto:Nick.Couchman@seakr.com] Sent: Thursday, October 28, 2010 12:23 PM To: Han, Weidong Cc: xen-devel@lists.xensource.com; konrad.wilk@Oracle.Com Subject: Re: [Xen-devel] PCI Passthrough Problems/Questions> the log of xm dmesg looks no problem. you''d better post complete log > and lspci -v.Sorry about that, xend.log is attached, and here''s the output of lspci -v: 00:00.0 Host bridge: Intel Corporation 4 Series Chipset DRAM Controller (rev 03) Subsystem: Dell Device 0420 Flags: bus master, fast devsel, latency 0 Capabilities: [e0] Vendor Specific Information: Len=0c Kernel driver in use: agpgart-intel 00:01.0 PCI bridge: Intel Corporation 4 Series Chipset PCI Express Root Port (rev 03) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=01, subordinate=02, sec-latency=0 I/O behind bridge: 0000d000-0000dfff Memory behind bridge: f7900000-f79fffff Prefetchable memory behind bridge: 00000000f0000000-00000000f00fffff Capabilities: [88] Subsystem: Dell Device 0420 Capabilities: [80] Power Management version 3 Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit- Capabilities: [a0] Express Root Port (Slot+), MSI 00 Capabilities: [100] Virtual Channel Capabilities: [140] Root Complex Link Kernel driver in use: pcieport 00:02.0 VGA compatible controller: Intel Corporation 4 Series Chipset Integrated Graphics Controller (rev 03) (prog-if 00 [VGA controller]) Subsystem: Dell Device 0420 Flags: bus master, fast devsel, latency 0, IRQ 435 Memory at f7c00000 (64-bit, non-prefetchable) [size=4M] Memory at e0000000 (64-bit, prefetchable) [size=256M] I/O ports at ecb0 [size=8] Expansion ROM at [disabled] Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit- Capabilities: [d0] Power Management version 2 Kernel driver in use: i915 00:02.1 Display controller: Intel Corporation 4 Series Chipset Integrated Graphics Controller (rev 03) Subsystem: Dell Device 0420 Flags: bus master, fast devsel, latency 0 Memory at f7b00000 (64-bit, non-prefetchable) [size=1M] Capabilities: [d0] Power Management version 2 00:03.0 Communication controller: Intel Corporation 4 Series Chipset HECI Controller (rev 03) Subsystem: Dell Device 0420 Flags: bus master, fast devsel, latency 0, IRQ 11 Memory at feda6000 (64-bit, non-prefetchable) [size=16] Capabilities: [50] Power Management version 3 Capabilities: [8c] MSI: Enable- Count=1/1 Maskable- 64bit+ 00:03.2 IDE interface: Intel Corporation 4 Series Chipset PT IDER Controller (rev 03) (prog-if 85 [Master SecO PriO]) Subsystem: Dell Device 0420 Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 9 I/O ports at fe80 [size=8] I/O ports at fe90 [size=4] I/O ports at fea0 [size=8] I/O ports at feb0 [size=4] I/O ports at fef0 [size=16] Capabilities: [c8] Power Management version 3 Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+ 00:03.3 Serial controller: Intel Corporation 4 Series Chipset Serial KT Controller (rev 03) (prog-if 02 [16550]) Subsystem: Dell Device 0420 Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 17 I/O ports at ecb8 [size=8] Memory at f7ad8000 (32-bit, non-prefetchable) [size=4K] Capabilities: [c8] Power Management version 3 Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+ Kernel driver in use: serial 00:19.0 Ethernet controller: Intel Corporation 82567LM-3 Gigabit Network Connection (rev 02) Subsystem: Dell Device 0276 Flags: bus master, fast devsel, latency 0, IRQ 434 Memory at f7ae0000 (32-bit, non-prefetchable) [size=128K] Memory at f7ad9000 (32-bit, non-prefetchable) [size=4K] I/O ports at ece0 [size=32] Capabilities: [c8] Power Management version 2 Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [e0] PCI Advanced Features Kernel driver in use: e1000e 00:1a.0 USB Controller: Intel Corporation 82801JD/DO (ICH10 Family) USB UHCI Controller #4 (rev 02) (prog-if 00 [UHCI]) Subsystem: Dell Device 0420 Flags: bus master, medium devsel, latency 0, IRQ 16 I/O ports at ff20 [size=32] Capabilities: [50] PCI Advanced Features Kernel driver in use: uhci_hcd 00:1a.1 USB Controller: Intel Corporation 82801JD/DO (ICH10 Family) USB UHCI Controller #5 (rev 02) (prog-if 00 [UHCI]) Subsystem: Dell Device 0420 Flags: bus master, medium devsel, latency 0, IRQ 17 I/O ports at ff00 [size=32] Capabilities: [50] PCI Advanced Features Kernel driver in use: uhci_hcd 00:1a.2 USB Controller: Intel Corporation 82801JD/DO (ICH10 Family) USB UHCI Controller #6 (rev 02) (prog-if 00 [UHCI]) Subsystem: Dell Device 0420 Flags: bus master, medium devsel, latency 0, IRQ 22 I/O ports at fc00 [size=32] Capabilities: [50] PCI Advanced Features Kernel driver in use: uhci_hcd 00:1a.7 USB Controller: Intel Corporation 82801JD/DO (ICH10 Family) USB2 EHCI Controller #2 (rev 02) (prog-if 20 [EHCI]) Subsystem: Dell Device 0420 Flags: bus master, medium devsel, latency 0, IRQ 22 Memory at f7ada000 (32-bit, non-prefetchable) [size=1K] Capabilities: [50] Power Management version 2 Capabilities: [58] Debug port: BAR=1 offset=00a0 Capabilities: [98] PCI Advanced Features Kernel driver in use: ehci_hcd 00:1b.0 Audio device: Intel Corporation 82801JD/DO (ICH10 Family) HD Audio Controller (rev 02) Subsystem: Dell Device 0420 Flags: bus master, fast devsel, latency 0, IRQ 433 Memory at f7adc000 (64-bit, non-prefetchable) [size=16K] Capabilities: [50] Power Management version 2 Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00 Capabilities: [100] Virtual Channel Capabilities: [130] Root Complex Link Kernel driver in use: HDA Intel 00:1c.0 PCI bridge: Intel Corporation 82801JD/DO (ICH10 Family) PCI Express Port 1 (rev 02) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=03, subordinate=03, sec-latency=0 I/O behind bridge: 00001000-00001fff Memory behind bridge: f7800000-f78fffff Prefetchable memory behind bridge: 00000000fc000000-00000000fc1fffff Capabilities: [40] Express Root Port (Slot+), MSI 00 Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit- Capabilities: [90] Subsystem: Dell Device 0420 Capabilities: [a0] Power Management version 2 Capabilities: [100] Virtual Channel Capabilities: [180] Root Complex Link Kernel driver in use: pcieport 00:1c.1 PCI bridge: Intel Corporation 82801JD/DO (ICH10 Family) PCI Express Port 2 (rev 02) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=04, subordinate=04, sec-latency=0 I/O behind bridge: 00002000-00002fff Memory behind bridge: f7700000-f77fffff Prefetchable memory behind bridge: 00000000fc200000-00000000fc3fffff Capabilities: [40] Express Root Port (Slot+), MSI 00 Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit- Capabilities: [90] Subsystem: Dell Device 0420 Capabilities: [a0] Power Management version 2 Capabilities: [100] Virtual Channel Capabilities: [180] Root Complex Link Kernel driver in use: pcieport 00:1d.0 USB Controller: Intel Corporation 82801JD/DO (ICH10 Family) USB UHCI Controller #1 (rev 02) (prog-if 00 [UHCI]) Subsystem: Dell Device 0420 Flags: bus master, medium devsel, latency 0, IRQ 23 I/O ports at ff80 [size=32] Capabilities: [50] PCI Advanced Features Kernel driver in use: uhci_hcd 00:1d.1 USB Controller: Intel Corporation 82801JD/DO (ICH10 Family) USB UHCI Controller #2 (rev 02) (prog-if 00 [UHCI]) Subsystem: Dell Device 0420 Flags: bus master, medium devsel, latency 0, IRQ 17 I/O ports at ff60 [size=32] Capabilities: [50] PCI Advanced Features Kernel driver in use: uhci_hcd 00:1d.2 USB Controller: Intel Corporation 82801JD/DO (ICH10 Family) USB UHCI Controller #3 (rev 02) (prog-if 00 [UHCI]) Subsystem: Dell Device 0420 Flags: bus master, medium devsel, latency 0, IRQ 18 I/O ports at ff40 [size=32] Capabilities: [50] PCI Advanced Features Kernel driver in use: uhci_hcd 00:1d.7 USB Controller: Intel Corporation 82801JD/DO (ICH10 Family) USB2 EHCI Controller #1 (rev 02) (prog-if 20 [EHCI]) Subsystem: Dell Device 0420 Flags: bus master, medium devsel, latency 0, IRQ 23 Memory at ff980000 (32-bit, non-prefetchable) [size=1K] Capabilities: [50] Power Management version 2 Capabilities: [58] Debug port: BAR=1 offset=00a0 Capabilities: [98] PCI Advanced Features Kernel driver in use: ehci_hcd 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a2) (prog-if 01 [Subtractive decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=05, subordinate=05, sec-latency=32 Capabilities: [50] Subsystem: Dell Device 0420 00:1f.0 ISA bridge: Intel Corporation 82801JDO (ICH10DO) LPC Interface Controller (rev 02) Subsystem: Dell Device 0420 Flags: bus master, medium devsel, latency 0 Capabilities: [e0] Vendor Specific Information: Len=0c 00:1f.2 RAID bus controller: Intel Corporation 82801 SATA RAID Controller (rev 02) Subsystem: Dell Device 0420 Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 436 I/O ports at fe00 [size=8] I/O ports at fe10 [size=4] I/O ports at fe20 [size=8] I/O ports at fe30 [size=4] I/O ports at fec0 [size=32] Memory at ff970000 (32-bit, non-prefetchable) [size=2K] Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit- Capabilities: [70] Power Management version 3 Capabilities: [a8] SATA HBA v1.0 Capabilities: [b0] PCI Advanced Features Kernel driver in use: ahci 00:1f.3 SMBus: Intel Corporation 82801JD/DO (ICH10 Family) SMBus Controller (rev 02) Subsystem: Dell Device 0420 Flags: medium devsel, IRQ 18 Memory at f7adb000 (64-bit, non-prefetchable) [size=256] I/O ports at 0980 [size=32] Kernel driver in use: i801_smbus 01:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Memory at f0000000 (64-bit, prefetchable) [size=64K] Bus: primary=01, secondary=02, subordinate=02, sec-latency=64 I/O behind bridge: 0000d000-0000dfff Memory behind bridge: f7900000-f79fffff Capabilities: [40] Power Management version 2 Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [60] Express PCI/PCI-X Bridge, MSI 00 Capabilities: [100] Power Budgeting Kernel driver in use: pciback 02:00.0 Serial controller: Perle Systems Ltd Device 0331 (prog-if 06 [16950]) Subsystem: Oxford Semiconductor Ltd Device 9501 Flags: medium devsel, IRQ 16 I/O ports at dc80 [disabled] [size=32] Memory at f79fc000 (32-bit, non-prefetchable) [disabled] [size=4K] I/O ports at dca0 [disabled] [size=32] Memory at f79fd000 (32-bit, non-prefetchable) [disabled] [size=4K] Capabilities: [40] Power Management version 1 Kernel driver in use: pciback 02:00.1 Bridge: Perle Systems Ltd Device 0331 Subsystem: Oxford Semiconductor Ltd Device 9511 Flags: medium devsel, IRQ 16 I/O ports at dcc0 [disabled] [size=32] Memory at f79fe000 (32-bit, non-prefetchable) [disabled] [size=4K] I/O ports at dce0 [disabled] [size=32] Memory at f79ff000 (32-bit, non-prefetchable) [disabled] [size=4K] Capabilities: [40] Power Management version 1 Kernel driver in use: pciback and lspci -vt: -[0000:00]-+-00.0 Intel Corporation 4 Series Chipset DRAM Controller +-01.0-[01-02]----00.0-[02]--+-00.0 Perle Systems Ltd Device 0331 | \-00.1 Perle Systems Ltd Device 0331 +-02.0 Intel Corporation 4 Series Chipset Integrated Graphics Controller +-02.1 Intel Corporation 4 Series Chipset Integrated Graphics Controller +-03.0 Intel Corporation 4 Series Chipset HECI Controller +-03.2 Intel Corporation 4 Series Chipset PT IDER Controller +-03.3 Intel Corporation 4 Series Chipset Serial KT Controller +-19.0 Intel Corporation 82567LM-3 Gigabit Network Connection +-1a.0 Intel Corporation 82801JD/DO (ICH10 Family) USB UHCI Controller #4 +-1a.1 Intel Corporation 82801JD/DO (ICH10 Family) USB UHCI Controller #5 +-1a.2 Intel Corporation 82801JD/DO (ICH10 Family) USB UHCI Controller #6 +-1a.7 Intel Corporation 82801JD/DO (ICH10 Family) USB2 EHCI Controller #2 +-1b.0 Intel Corporation 82801JD/DO (ICH10 Family) HD Audio Controller +-1c.0-[03]-- +-1c.1-[04]-- +-1d.0 Intel Corporation 82801JD/DO (ICH10 Family) USB UHCI Controller #1 +-1d.1 Intel Corporation 82801JD/DO (ICH10 Family) USB UHCI Controller #2 +-1d.2 Intel Corporation 82801JD/DO (ICH10 Family) USB UHCI Controller #3 +-1d.7 Intel Corporation 82801JD/DO (ICH10 Family) USB2 EHCI Controller #1 +-1e.0-[05]-- +-1f.0 Intel Corporation 82801JDO (ICH10DO) LPC Interface Controller +-1f.2 Intel Corporation 82801 SATA RAID Controller -Nick -------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> "Han, Weidong" 10/27/10 10:38 PM >>> > I don''t want xend.log. I want xen log (xm dmesg or captured by serial port). And pls also post qemu log (/var/log/xen/dm-qemu-xxx.log).Sorry about that, attached are the output from xm dmesg and the qemu-dm-XP.log file. -Nick -------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Nick Couchman wrote:>>>> "Han, Weidong" 10/27/10 10:38 PM >>> >>>> >> I don''t want xend.log. I want xen log (xm dmesg or captured by serial port). And pls also post qemu log (/var/log/xen/dm-qemu-xxx.log). >> > > Sorry about that, attached are the output from xm dmesg and the qemu-dm-XP.log file. > >qemu-dm-XP.log looks fine. Per "xm dmesg" log, unmap[ing and mapping 00:02.0 and 00:02.1 are no problem. But why are there many unmap and map (d1 -> d5)? did you create guest 5 times? Could you describe the crash you encountered? BTW, the "xm dmesg" log looks not complete, it should include following logs when you create hvm guest: (XEN) HVM1: HVM Loader (XEN) HVM1: Detected Xen v4.0.0-rc3-pre (XEN) HVM1: CPU speed is 2533 MHz ... and pls add "iommu=verbose" in grub to get more VT-d info. Regards, Weidong _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> qemu-dm-XP.log looks fine. Per "xm dmesg" log, unmap[ing and mapping > 00:02.0 and 00:02.1 are no problem. But why are there many unmap and map > (d1 -> d5)? did you create guest 5 times? Could you describe the crash > you encountered? BTW, the "xm dmesg" log looks not complete, it should > include following logs when you create hvm guest: > (XEN) HVM1: HVM Loader > (XEN) HVM1: Detected Xen v4.0.0-rc3-pre > (XEN) HVM1: CPU speed is 2533 MHz > ... > > and pls add "iommu=verbose" in grub to get more VT-d info. > > Regards, > WeidongWeidong, First, I have the following already specified on the xen.gz kernel line in grub: loglevel=all guest_lvl=all iommu=verbose Yes, I had started the guest five times...I''ve rebooted the system and started it once, and have attached the log files. As far as the domU crashing - there''s not much to describe. There is no longer any error after issuing XM start, but the domU doesn''t ever get to the point of displaying anything on the screen - it crashes almost as soon as the xm start command is issued (right after doing "xm start" the "xm list" command shows the domU not running). So, the log file is going to be the only form of output I can get you - there''s nothing to report from the screen. I don''t see the output you mentioned anywhere in xm dmesg. -Nick -------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Nick Couchman wrote:>> qemu-dm-XP.log looks fine. Per "xm dmesg" log, unmap[ing and mapping >> 00:02.0 and 00:02.1 are no problem. But why are there many unmap and map >> (d1 -> d5)? did you create guest 5 times? Could you describe the crash >> you encountered? BTW, the "xm dmesg" log looks not complete, it should >> include following logs when you create hvm guest: >> (XEN) HVM1: HVM Loader >> (XEN) HVM1: Detected Xen v4.0.0-rc3-pre >> (XEN) HVM1: CPU speed is 2533 MHz >> ... >> >> and pls add "iommu=verbose" in grub to get more VT-d info. >> >> Regards, >> Weidong >> > > Weidong, > First, I have the following already specified on the xen.gz kernel line > in grub: > > loglevel=all guest_lvl=all iommu=verbose > > Yes, I had started the guest five times...I''ve rebooted the system and > started it once, and have attached the log files. As far as the domU > crashing - there''s not much to describe. There is no longer any error > after issuing XM start, but the domU doesn''t ever get to the point of > displaying anything on the screen - it crashes almost as soon as the xm > start command is issued (right after doing "xm start" the "xm list" > command shows the domU not running). So, the log file is going to be > the only form of output I can get you - there''s nothing to report from > the screen. I don''t see the output you mentioned anywhere in xm dmesg. > > -Nick > > >The logs look no problem. I''m wondering if the device implies PCI standard. At last, I think you can try: only assign 02:00.0 (pci=[02:00.0]), or assign them as multiple devices in guest (pci=[02:00.0-1]). Regards, Weidong> -------- > > This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > > > > > > The logs look no problem. I''m wondering if the device implies PCI > standard. At last, I think you can try: only assign 02:00.0 > (pci=[02:00.0]), or assign them as multiple devices in guest > (pci=[02:00.0-1]). > > Regards, > WeidongShould have tried that on my own. Forwarding just 02:00.0 through worked just fine! So, the patch you had me apply to the Xen 4 source - is that a final version of the patch, or is there still some work to do to come up with a final fix for this? -Nick -------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Nick Couchman wrote:>>> >>> >> The logs look no problem. I''m wondering if the device implies PCI >> standard. At last, I think you can try: only assign 02:00.0 >> (pci=[02:00.0]), or assign them as multiple devices in guest >> (pci=[02:00.0-1]). >> >> Regards, >> Weidong >> > > Should have tried that on my own. Forwarding just 02:00.0 through > worked just fine! > > So, the patch you had me apply to the Xen 4 source - is that a final > version of the patch, or is there still some work to do to come up with > a final fix for this? > >The patch is just for your testing. I think Jan''s patch should fix the issue. Jan sent it out in xen-devel mailing list yesterday, I attached it here, pls try with it. Regards, Weidong _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I can confirm that Jan''s patch fixes the problem! Thanks for the help! -Nick On Wed, 2010-11-03 at 09:01 +0800, Weidong Han wrote:> Nick Couchman wrote: > >>> > >>> > >> The logs look no problem. I''m wondering if the device implies PCI > >> standard. At last, I think you can try: only assign 02:00.0 > >> (pci=[02:00.0]), or assign them as multiple devices in guest > >> (pci=[02:00.0-1]). > >> > >> Regards, > >> Weidong > >> > > > > Should have tried that on my own. Forwarding just 02:00.0 through > > worked just fine! > > > > So, the patch you had me apply to the Xen 4 source - is that a final > > version of the patch, or is there still some work to do to come up with > > a final fix for this? > > > > > The patch is just for your testing. I think Jan''s patch should fix the > issue. Jan sent it out in xen-devel mailing list yesterday, I attached > it here, pls try with it. > > Regards, > Weidong > plain text document attachment (vtd-check-secbus-devfn.patch) > If the device at <secbus>:00.0 is the device the mapping operation was > initiated for, trying to map it a second time will fail, and hence > this second mapping attempt must be prevented (as was done prior to > said c/s). > > Once at it, simplify the code a little, too. > > Signed-off-by: Jan Beulich <jbeulich@novell.com> > > --- a/xen/drivers/passthrough/vtd/iommu.c > +++ b/xen/drivers/passthrough/vtd/iommu.c > @@ -1371,23 +1371,16 @@ static int domain_context_mapping(struct > if ( find_upstream_bridge(&bus, &devfn, &secbus) < 1 ) > break; > > - /* PCIe to PCI/PCIx bridge */ > - if ( pdev_type(bus, devfn) == DEV_TYPE_PCIe2PCI_BRIDGE ) > - { > - ret = domain_context_mapping_one(domain, drhd->iommu, bus, devfn); > - if ( ret ) > - return ret; > + ret = domain_context_mapping_one(domain, drhd->iommu, bus, devfn); > > - /* > - * Devices behind PCIe-to-PCI/PCIx bridge may generate > - * different requester-id. It may originate from devfn=0 > - * on the secondary bus behind the bridge. Map that id > - * as well. > - */ > + /* > + * Devices behind PCIe-to-PCI/PCIx bridge may generate different > + * requester-id. It may originate from devfn=0 on the secondary bus > + * behind the bridge. Map that id as well if we didn''t already. > + */ > + if ( !ret && pdev_type(bus, devfn) == DEV_TYPE_PCIe2PCI_BRIDGE && > + (secbus != pdev->bus || pdev->devfn != 0) ) > ret = domain_context_mapping_one(domain, drhd->iommu, secbus, 0); > - } > - else /* Legacy PCI bridge */ > - ret = domain_context_mapping_one(domain, drhd->iommu, bus, devfn); > > break; >-------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel