I just wanted to drop a note to say that I have managed to get VGA passthrough working with multiple cards and domUs. The cards I am using are quasi Quadro 5000 (modified GTX470) and quasi Quadro 6000 (modified GTX480). (Here is an older article about modified GTS450 modified to Quadro 2000 if anyone is interested: http://www.altechnative.net/2013/06/23/nvidia-cards-geforce-quadro-and-geforce-modified-into-a-quadro-for-virtualized-gaming/ ) This succeeded despite all the PCIe slots being behind NF200 bridges without ACS support. I now have two VMs with VGA passthrough running, with one card passed through to each one. The OS is EL6, and I am using Xen packages and kernel from the CRC repository. What is strange, however, is that whereas I had issues with a single VM crashing on reboot with VGA passthrough (and usually taking the host down with it), with two VMs running, I seem to be able to reboot them dozens of times without any crashes, problems, or performance degradation (yes, rebooting VMs with VGA passthrough without rebooting the host and there appears to be no effect on performance with Nvidia Quadro cards). This new found stability could be down to some of the PCI changes, though, with different USB controllers being passed through, and NIC being bridged rather than PCI passthrough since there are now two domUs that have to share it, so it is possible that the PCI memory stomp I am suspecting to be the root cause is getting coincidentally avoided (not that I''m complaining!). I''m happy to share more details on and off list if anybody is interested. Gordan
Hello Gordan, I would like to hear more, any chance you can share the xen version, and whether you applied any patches? I know you were looking into the primary passthrough patch and also MMIO (in relation to memory stomping). Did you apply those patches and are you using upstream qemu and/or primary passthrough? Any tips on applying the patches would also be great. Thanks, ~Casey On Sat, Jul 6, 2013 at 5:11 PM, Gordan Bobic <gordan@bobich.net> wrote:> I just wanted to drop a note to say that I have managed to get VGA > passthrough working with multiple cards and domUs. > > The cards I am using are quasi Quadro 5000 (modified GTX470) and quasi > Quadro 6000 (modified GTX480). (Here is an older article about modified > GTS450 modified to Quadro 2000 if anyone is interested: > http://www.altechnative.net/**2013/06/23/nvidia-cards-** > geforce-quadro-and-geforce-**modified-into-a-quadro-for-** > virtualized-gaming/<http://www.altechnative.net/2013/06/23/nvidia-cards-geforce-quadro-and-geforce-modified-into-a-quadro-for-virtualized-gaming/>) > > This succeeded despite all the PCIe slots being behind NF200 bridges > without ACS support. > > I now have two VMs with VGA passthrough running, with one card passed > through to each one. The OS is EL6, and I am using Xen packages and kernel > from the CRC repository. > > What is strange, however, is that whereas I had issues with a single VM > crashing on reboot with VGA passthrough (and usually taking the host down > with it), with two VMs running, I seem to be able to reboot them dozens of > times without any crashes, problems, or performance degradation (yes, > rebooting VMs with VGA passthrough without rebooting the host and there > appears to be no effect on performance with Nvidia Quadro cards). This new > found stability could be down to some of the PCI changes, though, with > different USB controllers being passed through, and NIC being bridged > rather than PCI passthrough since there are now two domUs that have to > share it, so it is possible that the PCI memory stomp I am suspecting to be > the root cause is getting coincidentally avoided (not that I''m > complaining!). > > I''m happy to share more details on and off list if anybody is interested. > > Gordan > > ______________________________**_________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On Sun, 7 Jul 2013 01:28:45 -0700, Casey DeLorme <cdelorme@gmail.com> wrote:> Hello Gordan, > > I would like to hear more, any chance you can share the xen version, > and whether you applied any patches?As mentioned in the original email, I'm running on EL6 (Scientific) and I'm running the latest rpms of kernel and xen packages from the CRC repository (kernel 3.9.9, xen 4.2.2 with up to date XSA patches applied).> I know you were looking into > the primary passthrough patch and also MMIO (in relation to memory > stomping). Did you apply those patches and are you using upstream > qemu and/or primary passthrough? Any tips on applying the patches > would also be great.I have not applied any extra patches. The MMIO patch I didn't bother with - it involves statically defining BAR regions, so until I have time to write a better patch that auto-detects BARs at runtime and maps them all, I left it out. It might help with stability in some cases and avoid the PCI memory stomp, but I seem to have gotten this stable without it. I'll probably still do a dynamic pBAR=vBAR patch at some point because I'm hoping it'll mean I don't have to doctor my GeForce cards into Quadros to get them to work (and get working PhysX as a bonus since Quadro drivers don't seem to enable it). But for now all my GeForces are modified into Quadros, so until upgrade time, my motivation is running low. How long those GTX480->Quadro 6000 cards can keep up remains to be seen. :) The other part of the Nvidia primary passthrough patch is the BIOS initializer, which is currently limited to 64KB BIOSes. All but one GTX680 cards (a Gigabyte one) and all Titan/7xx cards and most other 6xx cards have UEFI in BIOS which bloats them past 64KB. This can be stripped out (done it, needs more testing before I'm 100% confident I didn't break anything - a weekend of various benchmarking ought to do it). But that requires card modifying, and if I've got to do that anyway, it kind of reduces the appeal of the patch, and Quadro drivers no doubt handle a few extra edge cases more gracefully. Gordan> On Sat, Jul 6, 2013 at 5:11 PM, Gordan Bobic wrote: > I just wanted to drop a note to say that I have managed to get VGA > passthrough working with multiple cards and domUs. > > The cards I am using are quasi Quadro 5000 (modified GTX470) and > quasi Quadro 6000 (modified GTX480). (Here is an older article about > modified GTS450 modified to Quadro 2000 if anyone is interested: > > > http://www.altechnative.net/2013/06/23/nvidia-cards-geforce-quadro-and-geforce-modified-into-a-quadro-for-virtualized-gaming/ > [2] ) > > This succeeded despite all the PCIe slots being behind NF200 bridges > without ACS support. > > I now have two VMs with VGA passthrough running, with one card > passed > through to each one. The OS is EL6, and I am using Xen packages and > kernel from the CRC repository. > > What is strange, however, is that whereas I had issues with a single > VM crashing on reboot with VGA passthrough (and usually taking the > host down with it), with two VMs running, I seem to be able to reboot > them dozens of times without any crashes, problems, or performance > degradation (yes, rebooting VMs with VGA passthrough without > rebooting > the host and there appears to be no effect on performance with Nvidia > Quadro cards). This new found stability could be down to some of the > PCI changes, though, with different USB controllers being passed > through, and NIC being bridged rather than PCI passthrough since > there > are now two domUs that have to share it, so it is possible that the > PCI memory stomp I am suspecting to be the root cause is getting > coincidentally avoided (not that I'm complaining!). > > I'm happy to share more details on and off list if anybody is > interested. > > Gordan > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org [3] > http://lists.xen.org/xen-users [4] > > > > Links: > ------ > [1] mailto:gordan@bobich.net > [2] > > http://www.altechnative.net/2013/06/23/nvidia-cards-geforce-quadro-and-geforce-modified-into-a-quadro-for-virtualized-gaming/ > [3] mailto:Xen-users@lists.xen.org > [4] http://lists.xen.org/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Possibly Parallel Threads
- Is it possible to mod a Geforce GTX 560 graphics card into a Quadro?
- Is it possible to mod a Geforce GTX 560 graphics card into a Quadro?
- Primary VGA Passthrough Status
- VGA Passthrough of AMD FirePro W7000 to Windows 8: Not enough resources
- Xen VGA Passthrough - GTX 480 successfully quadrified to quadro 6000 (softmod) - more than 4GB of RAM for Win XP 64 Bits