Hello Everyone, My company is in the process of implementing Xen virtualization. We have managed to get Xen up and running, and have worked through all of the kinks in our set up, except for PCI passthrough. PCI passthrough doesn''t work, and I suspect it''s because Xen is not using the IOMMU in the computer, even though we built the computer with components that support VT-d. Here are the computer components we used: Motherboard: Intel BOXDH77DF CPU: Intel Core i5-2310 (Sandy Bridge - BX80623I52310) I also booted Xen up with the "iommu=verbose" parameter and ran "xl dmesg" on the computer, and uploaded the output here: http://pastebin.com/rF0kyftt. I tried running Xen with the "iommu=required" flag, and it failed because it said it couldn''t use the IOMMU. If someone work with us to figure out why Xen doesn''t use the IOMMU, it would be very appreciated. Thanks. Philip Wernersbach Software Implementation Division Jacobs Automation _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Philip Wernersbach
2013-May-15 19:06 UTC
Re: Xen Reports no IOMMU, but Hardware Supports It
Paul, Thanks for taking the time to help. The host operating system is NetBSD 6.0. There is one Windows XP guest, and the output of lspci -vvnn is at http://pastebin.com/zPNdnnjK. We are trying to pass through the USB controllers on the machine, which should be PCI addresses 00:14.0, 00:1a.0, and 00:1d.0. Thanks, Philip Wernersbach Software Implementation Division Jacobs Automation On Wed, May 15, 2013 at 3:49 AM, Paul Stimpson < paul.stimpson@redfoxdigital.com> wrote:> Hi Philip, > > I run a consultancy and I''m just working with a name-you-know company to > deliver a xenized appliance that uses IOMMU. > > We found that getting IOMMU working was one if the more challenging parts > of the project. > > Please will you tell me what the host operating system is (including the > version), how many guests there are, what the guests'' operating systems > are, what the piece of hardware you are trying to pass is and the output of > the command lspci? > > Best regards, > Paul. > > > Philip Wernersbach <pwernersbach@jacobsautomation.com> wrote: >> >> Hello Everyone, >> >> My company is in the process of implementing Xen virtualization. We have >> managed to get Xen up and running, and have worked through all of the kinks >> in our set up, except for PCI passthrough. PCI passthrough doesn''t work, >> and I suspect it''s because Xen is not using the IOMMU in the computer, even >> though we built the computer with components that support VT-d. >> >> Here are the computer components we used: >> Motherboard: Intel BOXDH77DF >> CPU: Intel Core i5-2310 (Sandy Bridge - BX80623I52310) >> >> I also booted Xen up with the "iommu=verbose" parameter and ran "xl >> dmesg" on the computer, and uploaded the output here: >> http://pastebin.com/rF0kyftt. I tried running Xen with the >> "iommu=required" flag, and it failed because it said it couldn''t use the >> IOMMU. If someone work with us to figure out why Xen doesn''t use the IOMMU, >> it would be very appreciated. >> >> Thanks. >> >> Philip Wernersbach >> Software Implementation Division >> Jacobs Automation >> >> ------------------------------ >> >> Xen-users mailing list >> Xen-users@lists.xen.org >> http://lists.xen.org/xen-users >> >> > -- > Sent from Kaiten Mail on Android. Please excuse my brevity. >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Philip Wernersbach
2013-May-15 19:15 UTC
Re: Xen Reports no IOMMU, but Hardware Supports It
Chris, There is no option for VT-d in the BIOS. However, there is an option for Intel Virtualization Technology (VT-x), which is enabled. Even though there''s no option for VT-d, according to [1], the board (DH77DF) supports VT-d. Thanks, Philip Wernersbach Software Implementation Division Jacobs Automation References: [1]: http://www.intel.com/support/motherboards/desktop/sb/CS-030922.htm On Wed, May 15, 2013 at 5:55 AM, chris <tknchris@gmail.com> wrote:> Did you enable VT-d in the bios? It must be enabled in the bios for it to > work > > On Tue, May 14, 2013 at 10:21 PM, Philip Wernersbach > <pwernersbach@jacobsautomation.com> wrote: > > Hello Everyone, > > > > My company is in the process of implementing Xen virtualization. We have > > managed to get Xen up and running, and have worked through all of the > kinks > > in our set up, except for PCI passthrough. PCI passthrough doesn''t work, > and > > I suspect it''s because Xen is not using the IOMMU in the computer, even > > though we built the computer with components that support VT-d. > > > > Here are the computer components we used: > > Motherboard: Intel BOXDH77DF > > CPU: Intel Core i5-2310 (Sandy Bridge - BX80623I52310) > > > > I also booted Xen up with the "iommu=verbose" parameter and ran "xl > dmesg" > > on the computer, and uploaded the output here: > http://pastebin.com/rF0kyftt. > > I tried running Xen with the "iommu=required" flag, and it failed > because it > > said it couldn''t use the IOMMU. If someone work with us to figure out why > > Xen doesn''t use the IOMMU, it would be very appreciated. > > > > Thanks. > > > > Philip Wernersbach > > Software Implementation Division > > Jacobs Automation > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xen.org > > http://lists.xen.org/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Hi Philip, We solved this problem recently. The issue we found was that the USB interfaces were being picked up and bound to the host OS before the pciback module had loaded. This meant that the oddball module was loading, trying to bind its configured devices and finding they were already taken. Which version of Xen are you running? We found we couldn''t make pass through work until we upgraded to 4.2.0. Also,.it may sound silly but check in the bios that all the virtualization features are enabled. Some of the boards we have come out the box with them on, some with them off. It seems to depend on whether the board was intended for a server or a workstation. Try this test (assuming you are running the xl toolkit - if not, modify the command to be appropriate for xm): Shut down the windows guest with xl shutdown guest_name Manually bind the PCI devices: xl pci-assignable-add 00:14.0 xl pci-assignable-add 00:1a.0 xl pci-assignable-add 00:1d.0 Restart the guest with xl create filename If the USB interfaces magically appear in Windows, this confirms that the y are getting bound to the host early. There are several ways to solve this issue. One is to rebuild the host kernel and compile the Xen stuff into it so pciback is in the kernel and gets there first. This, I believe, is the "proper" way to do it. The other I know of is to modify the xendomains script to pci-assignable-add the BDFs as above before starting the guests. This isn''t so nice but you don''t have to build a new kernel. Sorry for the bad layout of this email. Written on my phone with fat fingers on a moving train :-) Bests, Paul. Philip Wernersbach <pwernersbach@jacobsautomation.com> wrote:>Paul, > >Thanks for taking the time to help. The host operating system is NetBSD >6.0. There is one Windows XP guest, and the output of lspci -vvnn is >at >http://pastebin.com/zPNdnnjK. We are trying to pass through the USB >controllers on the machine, which should be PCI addresses 00:14.0, >00:1a.0, >and 00:1d.0. > >Thanks, > >Philip Wernersbach >Software Implementation Division >Jacobs Automation > > >On Wed, May 15, 2013 at 3:49 AM, Paul Stimpson < >paul.stimpson@redfoxdigital.com> wrote: > >> Hi Philip, >> >> I run a consultancy and I''m just working with a name-you-know company >to >> deliver a xenized appliance that uses IOMMU. >> >> We found that getting IOMMU working was one if the more challenging >parts >> of the project. >> >> Please will you tell me what the host operating system is (including >the >> version), how many guests there are, what the guests'' operating >systems >> are, what the piece of hardware you are trying to pass is and the >output of >> the command lspci? >> >> Best regards, >> Paul. >> >> >> Philip Wernersbach <pwernersbach@jacobsautomation.com> wrote: >>> >>> Hello Everyone, >>> >>> My company is in the process of implementing Xen virtualization. We >have >>> managed to get Xen up and running, and have worked through all of >the kinks >>> in our set up, except for PCI passthrough. PCI passthrough doesn''t >work, >>> and I suspect it''s because Xen is not using the IOMMU in the >computer, even >>> though we built the computer with components that support VT-d. >>> >>> Here are the computer components we used: >>> Motherboard: Intel BOXDH77DF >>> CPU: Intel Core i5-2310 (Sandy Bridge - BX80623I52310) >>> >>> I also booted Xen up with the "iommu=verbose" parameter and ran "xl >>> dmesg" on the computer, and uploaded the output here: >>> http://pastebin.com/rF0kyftt. I tried running Xen with the >>> "iommu=required" flag, and it failed because it said it couldn''t use >the >>> IOMMU. If someone work with us to figure out why Xen doesn''t use the >IOMMU, >>> it would be very appreciated. >>> >>> Thanks. >>> >>> Philip Wernersbach >>> Software Implementation Division >>> Jacobs Automation >>> >>> ------------------------------ >>> >>> Xen-users mailing list >>> Xen-users@lists.xen.org >>> http://lists.xen.org/xen-users >>> >>> >> -- >> Sent from Kaiten Mail on Android. Please excuse my brevity. >>-- Sent from Kaiten Mail on Android. Please excuse my brevity. ------MPH2C0KK35D0VCB9WRJJ5W7NYJZI18 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html><head></head><body><p dir="ltr">Hi Philip,</p> <p dir="ltr">We solved this problem recently. The issue we found was that the USB interfaces were being picked up and bound to the host OS before the pciback module had loaded. This meant that the oddball module was loading, trying to bind its configured devices and finding they were already taken. </p> <p dir="ltr">Which version of Xen are you running? We found we couldn''t make pass through work until we upgraded to 4.2.0. Also,.it may sound silly but check in the bios that all the virtualization features are enabled. Some of the boards we have come out the box with them on, some with them off. It seems to depend on whether the board was intended for a server or a workstation.</p> <p dir="ltr">Try this test (assuming you are running the xl toolkit - if not, modify the command to be appropriate for xm):</p> <p dir="ltr">Shut down the windows guest with xl shutdown guest_name<br> Manually bind the PCI devices: xl pci-assignable-add 00:14.0<br> xl pci-assignable-add 00:1a.0<br> xl pci-assignable-add 00:1d.0<br> Restart the guest with xl create filename</p> <p dir="ltr">If the USB interfaces magically appear in Windows, this confirms that the y are getting bound to the host early.</p> <p dir="ltr">There are several ways to solve this issue.</p> <p dir="ltr">One is to rebuild the host kernel and compile the Xen stuff into it so pciback is in the kernel and gets there first. This, I believe, is the "proper" way to do it.</p> <p dir="ltr">The other I know of is to modify the xendomains script to pci-assignable-add the BDFs as above before starting the guests. This isn''t so nice but you don''t have to build a new kernel.</p> <p dir="ltr">Sorry for the bad layout of this email. Written on my phone with fat fingers on a moving train :-) </p> <p dir="ltr">Bests,<br> Paul.</p> <br><br><div class="gmail_quote">Philip Wernersbach <pwernersbach@jacobsautomation.com> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"> <div dir="ltr">Paul,<div><br /></div><div style="style">Thanks for taking the time to help. The host operating system is NetBSD 6.0. There is one Windows XP guest, and the output of lspci -vvnn is at <a href="http://pastebin.com/zPNdnnjK">http://pastebin.com/zPNdnnjK</a>. We are trying to pass through the USB controllers on the machine, which should be PCI addresses 00:14.0, 00:1a.0, and 00:1d.0.</div> <div style="style"><br /></div><div style="style">Thanks,</div><div style="style"><br /></div><div style="style">Philip Wernersbach</div><div style="style">Software Implementation Division</div><div style="style">Jacobs Automation</div></div><div class="gmail_extra"><br /><br /> <div class="gmail_quote">On Wed, May 15, 2013 at 3:49 AM, Paul Stimpson <span dir="ltr"><<a href="mailto:paul.stimpson@redfoxdigital.com" target="_blank">paul.stimpson@redfoxdigital.com</a>></span> wrote:<br /><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div><p dir="ltr">Hi Philip,</p> <p dir="ltr">I run a consultancy and I'm just working with a name-you-know company to deliver a xenized appliance that uses IOMMU. </p> <p dir="ltr">We found that getting IOMMU working was one if the more challenging parts of the project.</p> <p dir="ltr">Please will you tell me what the host operating system is (including the version), how many guests there are, what the guests' operating systems are, what the piece of hardware you are trying to pass is and the output of the command lspci?</p> <p dir="ltr">Best regards,<br /> Paul.<br /> </p> <br /><br /><div class="gmail_quote">Philip Wernersbach <<a href="mailto:pwernersbach@jacobsautomation.com" target="_blank">pwernersbach@jacobsautomation.com</a>> wrote:<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div><div class="h5"> <div dir="ltr"><div>Hello Everyone,</div><div><br /></div><div>My company is in the process of implementing Xen virtualization. We have managed to get Xen up and running, and have worked through all of the kinks in our set up, except for PCI passthrough. PCI passthrough doesn't work, and I suspect it's because Xen is not using the IOMMU in the computer, even though we built the computer with components that support VT-d.</div> <div><br /></div><div>Here are the computer components we used:</div><div>Motherboard: Intel BOXDH77DF</div><div>CPU: Intel Core i5-2310 (Sandy Bridge - BX80623I52310)</div><div><br /></div><div> I also booted Xen up with the "iommu=verbose" parameter and ran "xl dmesg" on the computer, and uploaded the output here: <a href="http://pastebin.com/rF0kyftt" target="_blank">http://pastebin.com/rF0kyftt</a>. I tried running Xen with the "iommu=required" flag, and it failed because it said it couldn't use the IOMMU. If someone work with us to figure out why Xen doesn't use the IOMMU, it would be very appreciated.</div> <div><br /></div><div>Thanks.</div><div><br /></div><div>Philip Wernersbach</div><div><div style="font-family:arial,sans-serif;font-size:13px">Software Implementation Division</div><div style="font-family:arial,sans-serif;font-size:13px"> Jacobs Automation</div></div></div> <p style="margin-top:2.5em;margin-bottom:1em;border-bottom:1px solid #000"></p></div></div><pre><hr /><br />Xen-users mailing list<br /><a href="mailto:Xen-users@lists.xen.org" target="_blank">Xen-users@lists.xen.org</a><br /><a href="http://lists.xen.org/xen-users" target="_blank">http://lists.xen.org/xen-users</a></pre> </blockquote></div><span class="HOEnZb"><font color="#888888"><br /> -- <br /> Sent from Kaiten Mail on Android. Please excuse my brevity.</font></span></div></blockquote></div><br /></div> </blockquote></div><br> -- <br> Sent from Kaiten Mail on Android. Please excuse my brevity.</body></html> ------MPH2C0KK35D0VCB9WRJJ5W7NYJZI18-- --===============3398520994674884796=Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users --===============3398520994674884796==--
I think I am a bit later to the party. However, you said...> Here are the computer components we used:> Motherboard: Intel BOXDH77DF> CPU: Intel Core i5-2310 (Sandy Bridge - BX80623I52310)In case you still are stuck there, Core i5 2310 does NOT support VT-d: http://ark.intel.com/products/53445/ In Sandy Bridge, you needed at least a Core i5 2400: http://ark.intel.com/products/52207 Even if Motherboard supports it and all, you need a Processor enabled for it, too. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users