Is there anyone actually using VT-d for GPU cards? From what I understand NVidia is impossible, but AMD does work for some cards. I would appreciate any hints from people that are actively using this technology. -- Simon Toth
Simon, On Mon, Nov 18, 2013 at 8:00 AM, "Mgr. Šimon Tóth" <toth@fi.muni.cz> wrote:> Is there anyone actually using VT-d for GPU cards? > > From what I understand NVidia is impossible, but AMD does work for some > cards. I would appreciate any hints from people that are actively using > this technology. > > -- > Simon Toth > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users >I''m using it right now on my primary desktop/workstation at home - previously it was built with two graphics cards, a Radeon 6670 for dom0 and a Radeon 7750 for a Windows 7 domU. Recently I switched out the 7750 for a hardware modded Nvidia GTX 670 (changed PCI ID to a Grid K2, so can use the virtualization-aware drivers) and that has been working well for me. Regards, David _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On Mon, 18 Nov 2013 15:00:50 +0100, "Mgr. Šimon Tóth" <toth@fi.muni.cz> wrote:> Is there anyone actually using VT-d for GPU cards?Yes, many of us.> From what I understand NVidia is impossible,Not impossible. Quadro cards work perfectly. GeForce cards don't work at all, courtesy of Nvidia deciding GeForce users should not be allowed to have this working and making the driver not cooperate WRT the peculiarities of virtualization. GeForce cards can be modified into Quadro cards. The modified GeForce cards work perfectly virtualized. If you wish to go this route, there are at least two modified Fermi class (4xx series) GeForce cards on eBay at the moment going for a fraction of the cost of a real Quadro. I have used the Fermi class cards extensively, and have had absolutely no issues with them - they "just work". A number of people have tried modified Kepler class cards and the result has been that they generally work fine for most people but there have on occassion been reports of minor issues with these. If you are not too concerned about performance and you can make do with a GTX480 (i.e. you don't need to run Crysis at more than 1920x1200 with everything turned up to max), a modified one of those (into a Quadro 6000) is probably the most trouble free, most cost effective way to get a decently performing VGA passthrough setup going. If performance is not an issue, a GTS450 modified into a Quadro 2000 will do nicely. There was one of each of these on eBay as of this morning.> but AMD does work for > some cards. I would appreciate any hints from people that are > actively > using this technology.AMD/ATI sort-of works. Poorly. Almost. Anything newer than a HD4xxx card should "work" in this sense, but you rebooting the domU with the GPU passed through to will unpredictably result in one of the following: 1) Host (dom0) crash (complete system lock-up). I find this is the most common thing to happen, especially when ATI card is used as the primary console card. 2) domU crash at startup. This could be BSOD, or just the video output never coming up on either VNC or external monitor. 2.1) If you then try to xl destroy / xl create the same domU again it will result in 1) every time. 3) Fuzzy lines when dragging windows and massively degraded performance. I have only ever managed to achieve this level of "functioning" on a VM reboot with ATI cards that don't feature auxiliary power input (e.g. HD6450/HD7450). At the moment, I find ATI cards are NOT a recommendable solution. I am using a HD7970 for one of my VMs (modified GTX480s in all others) but only until I can perfect the hardware modification on my GTX690, at which point I will be very glad to see the back of the HD7970. The only reason I am not using one of the GTX480 cards in place of the HD7970 is because it wasn't quite managing a perfect frame rate with everything maxed out on my wife's 2560x1600 monitor in Borderlands 2. Note that modifying a Kepler class card properly requires professional skill - soldering 0402 (i.e. 1.5mm along the longest dimension) components manually takes specialist equipment unless you are an absolute ninja with a tiny soldering iron. Gordan _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Mgr. Šimon Tóth
2013-Nov-18 15:40 UTC
Re: Anyone using VT-d for GPUs on desktop/workstation?
> At the moment, I find ATI cards are NOT a recommendable > solution.That''s really bad news for me. As far as buying modified cards go, I could probably get a Quadro for less, due to shipping and import costs. But thanks for the info. -- Simon Toth
B "Mgr. Šimon Tóth" <toth@fi.muni.cz> wrote:>> At the moment, I find ATI cards are NOT a recommendable >> solution. > >That's really bad news for me. As far as buying modified cards go, I >could probably get a Quadro for less, due to shipping and import costs. > >But thanks for the info.I think it would more accurately be true to say that VGA passthrough is still rather bleeding edge and there are issues with many configurations. Xen does not help itself here, as it's very difficult to troubleshoot passthrough, although it's otherwise a decent virtualization solution. I only ever managed to get the Matrox G200e built into my S3210SHLC motherboard passed through in Xen and VMWare. Both a 6950 and 6450 were a complete failure and I did try a lot of configurations. KVM is a different matter. Alex Williams is highly motivated to improve passthrough and there's more debugging available. I've been able to get both the 6950 and 6450 passed through to a Linux VM using both fglrx and radeon drivers. I haven't managed it in Windows 7 yet. I'd personally prefer to use Xen for server virtualization above KVM - better tools and I like the idea of a type 1 hypervisor. My advice, if you want the least hassle and have the cash, is to find someone who has a completely solid solution and buy an identical system. Otherwise you're probably going to have issues.. PK _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On 11/22/2013 12:24 PM, Peter Kay wrote:> B > > "Mgr. Šimon Tóth" <toth@fi.muni.cz> wrote: >>> At the moment, I find ATI cards are NOT a recommendable >>> solution. >> >> That's really bad news for me. As far as buying modified cards go, I >> could probably get a Quadro for less, due to shipping and import costs. >> >> But thanks for the info. > > I think it would more accurately be true to say that VGA passthrough is still rather bleeding edge and there are issues with many configurations. Xen does not help itself here, as it's very difficult to troubleshoot passthrough, although it's otherwise a decent virtualization solution. > > I only ever managed to get the Matrox G200e built into my S3210SHLC motherboard passed through in Xen and VMWare. Both a 6950 and 6450 were a complete failure and I did try a lot of configurations. > > KVM is a different matter. Alex Williams is highly motivated to improve passthrough and there's more debugging available. I've been able to get both the 6950 and 6450 passed through to a Linux VM using both fglrx and radeon drivers. I haven't managed it in Windows 7 yet. > > I'd personally prefer to use Xen for server virtualization above KVM - better tools and I like the idea of a type 1 hypervisor. > > My advice, if you want the least hassle and have the cash, is to find someone who has a completely solid solution and buy an identical system. Otherwise you're probably going to have issues.. Having spent a lot of time/effort getting my setup working, I can summarize in the following 3 points: 1) Get a motherboard with properly functioning IOMMU; specifically, one that somebody here tried and confirmed to work. Perhaps we should have a wiki page for listing motherboards people have has success with, just like we have for GPUs. 1.1) Do NOT get a motherboard featuring Nvidia NF200 PCIe bridges. This can be made to work with some ugly hacks and patches, but it's best avoided, at least until PVH support is fully implemented (for e820_host option support). 2) Use an Nvidia Quadro GPU explicitly listed to support "MultiOS" (Quadro FXx800, Quadro [2456]000, Quadro Kx000). Or get a GeForce card and modify it (or see if there's a pre-modified one on eBay at the time). Gordan _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users