Patrick Proniewski
2013-Apr-01 21:28 UTC
PCI/VGA passthrough: differences between Xen and ESXi?
Hello, That''s my first post here. I''m absolutely newborn to Xen, never installed or used it. But I''m mastering many aspects of ESXi (vmare) and I''ve made some tests with PCI passthrough on this platform. My main goal is to create a powerful desktop workstation able to run simultaneously FreeBSD, some flavor of Linux (work/games), some flavor of Windows (games), Mac OS X (main OS). Everything is detailed here: <http://www.patpro.net/blog/index.php/2012/11/11/2200-escaping-the-apple-ecosystem-part-0/>. In order to achieve this goal, I must rely on PCI passthrough. At least VGA to get proper GPU power (3D gaming like Left 4 Dead 2, photoshop processing - I''m also a photograph), and sound. I was able to achieve very good performance running Windows on top of ESXi on my Mac Pro, with stock ATI Radeon passed-through, even better than native Mac OS X performances :). I''ve ended my testing with ESXi, because I was not able to passthrough sound device, and USB. But PCI Passthrough on ESXi comes with major limitations: - memory allocation is fully reserved (ie. can''t start this VM if all its allocated memory is not available) - no snapshot possible - no sleep - one PCI device can only be passed-through to one VM - and some others... I''ve read the documentation for Xen about VGA and PCI passthrough, but I''m totally new to a great part of Xen''s vocabulary, and I''m not sure to understand everything. It looks like Xen shares some limitations with ESXi. Xen''s Wiki reads "Xen 4.0.0 supports VGA passthrough of the primary graphics adapter (...) only". What about 4.2? Is it possible to create a virtualized desktop with VGA and PCI passthrough for 2 or 3 VM''s running simultaneously (ie. each one with its own video card)? regards, Patrick (i''ve subscribed to digest, Cc me, please)
David TECHER
2013-Apr-02 16:54 UTC
Re: PCI/VGA passthrough: differences between Xen and ESXi?
Patrick, 1) Xen''s vocabulary dom0 = the main (first) hypervisor, hosting all virtual machines -- like your Mac OS X domU = any virtual machine. This is not the real meaning but here I try to stay "understandable". But for better understanding please refer to http://en.wikipedia.org/wiki/Xen My understanding is that your dom0 has to be replaced by Linux so you can run Xen. But for this part I am not a Mac OS X expert. 2) More info - details listed beow are only available for a Xen dom0 running on Linux. [Extracted]:....I''ve ended my testing with ESXi, because I was not able to passthrough sound device, and USB.... [Comment]: With Xen you can :) [Extracted]: ...with stock ATI Radeon..; [Comment]...Xen offers VGA PassThrough feature for ATI Radeon. Here is a link to a Youtube video to run Crysis 3 on a domU Windows 7 64 Bits with HD 7970 http://www.youtube.com/watch?v=GTnchEG4YtI&feature=player_embedded As you can see - I can use PCI Passthrough to add a XBOX 360 Paddle while playing Crysis 3 (see at 09:12 in the Video) - I have sound too :) But it works on Linux domU too. Here is another Youtube video link http://www.youtube.com/watch?feature=player_embedded&v=KzqOIMaBgX0 But you need to apply a few patches to able this feature with Xen 4.2 and over. Not very complicated to do. Here are my instructions to do it but I know that there are currently better way to test it http://www.davidgis.fr/blog/index.php?2013/03/13/935-xen-43-vga-passthrough-ati-card-hd-7970-changeset-26706 [Extracted]...memory allocation [Comment]...Xen - limited to 3GB for RAM. A few developers are working on overpassing this limitation. [Extracted]...no snaphost possible [Comment]...If you used Xen with LVM is then snahshotting is easier to manage ;) [Extracted] no sleep [Comment] works on Xen for both Linux and Windows. The main issue is that you can reboot a domU without restarting the dom0. [Extracted] Is it possible to create a virtualized desktop with VGA and PCI passthrough for 2 or 3 VM''s running simultaneously (ie. each one with its own video card)? [Comment] Yes it is doable. Never tested but from experience share with other Xen users I know that it should be run fine. Hope it helps. Kind regards. David. ________________________________ De : Patrick Proniewski <patpro@patpro.net> À : xen-users@lists.xen.org Envoyé le : Lundi 1 avril 2013 23h28 Objet : [Xen-users] PCI/VGA passthrough: differences between Xen and ESXi? Hello, That''s my first post here. I''m absolutely newborn to Xen, never installed or used it. But I''m mastering many aspects of ESXi (vmare) and I''ve made some tests with PCI passthrough on this platform. My main goal is to create a powerful desktop workstation able to run simultaneously FreeBSD, some flavor of Linux (work/games), some flavor of Windows (games), Mac OS X (main OS). Everything is detailed here: <http://www.patpro.net/blog/index.php/2012/11/11/2200-escaping-the-apple-ecosystem-part-0/>. In order to achieve this goal, I must rely on PCI passthrough. At least VGA to get proper GPU power (3D gaming like Left 4 Dead 2, photoshop processing - I''m also a photograph), and sound. I was able to achieve very good performance running Windows on top of ESXi on my Mac Pro, with stock ATI Radeon passed-through, even better than native Mac OS X performances :). I''ve ended my testing with ESXi, because I was not able to passthrough sound device, and USB. But PCI Passthrough on ESXi comes with major limitations: - memory allocation is fully reserved (ie. can''t start this VM if all its allocated memory is not available) - no snapshot possible - no sleep - one PCI device can only be passed-through to one VM - and some others... I''ve read the documentation for Xen about VGA and PCI passthrough, but I''m totally new to a great part of Xen''s vocabulary, and I''m not sure to understand everything. It looks like Xen shares some limitations with ESXi. Xen''s Wiki reads "Xen 4.0.0 supports VGA passthrough of the primary graphics adapter (...) only". What about 4.2? Is it possible to create a virtualized desktop with VGA and PCI passthrough for 2 or 3 VM''s running simultaneously (ie. each one with its own video card)? regards, Patrick (i''ve subscribed to digest, Cc me, please) _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Alexandre Kouznetsov
2013-Apr-02 17:18 UTC
Re: PCI/VGA passthrough: differences between Xen and ESXi?
Hello. A little correction, if I may. El 02/04/13 10:54, David TECHER escribió:> dom0 = the main (first) hypervisor, hosting all virtual machines -- like > your Mac OS X > domU = any virtual machine.Dom0 is not the hypervisor. The way Xen works, all OS''s or machines running on the host are "Domains", and they run over a layer they call hypervisor. The hypervisor is a very low level piece of software between the actual OS''s and the hardware. The difference between Dom0 and the rest of DomUs is that Dom0 is "privileged": hypervisor accepts system calls from this virtual machine and allows it to access the hardware directly. Beside that, there is no much technical difference between Dom0 and DomU. The practice is to assign Dom0 very few resources dedicated and is used almost exclusively to control the hypervisor (start/stop VM''s, etc.) and access to storage and networking. Anything actually useful the computer do is done within a DomU. There are some restrictions about what OS may run on Dom0: It has to be paravirtualized, it has to run a kernel specifically supporting Dom0 operation, it need to have a toolkit to control the hypervisor (xe, xl or xm). Specifically this are the restrictions that would prevent to run a MacOS X, for example, on Dom0: Xen hypervisor would not understand it. Greetings. -- Alexandre Kouznetsov
Patrick Proniewski
2013-Apr-02 18:23 UTC
Re: PCI/VGA passthrough: differences between Xen and ESXi?
David, thanks for your answer, my remarks below: On 02 avr. 2013, at 18:54, David TECHER wrote:> Patrick, > > 1) Xen''s vocabulary > > dom0 = the main (first) hypervisor, hosting all virtual machines -- like your Mac OS X > domU = any virtual machine.ok, I''ve pretty much got that.> My understanding is that your dom0 has to be replaced by Linux so you can run Xen. But for this part I am not a Mac OS X expert.Exactly. Bare metal would be nice, but ESXi is nothing more than a very slim linux, so a full linux+Xen is not a problem.> 2) More info - details listed beow are only available for a Xen dom0 running on Linux. > > [Extracted]:....I''ve ended my testing with ESXi, because I was not able to passthrough sound device, and USB.... > [Comment]: With Xen you can :)with VMWare you can too, but the hardware of the Mac Pro is not fully supported. It looks like a dedicated (PCI) sound card and USB controller would be easy to passthrough. Nonetheless, using ESXi it would probably require a dedicated card (sound and usb) for every VM I want passthrough in. That would make many cards.> [Extracted]: ...with stock ATI Radeon..; > [Comment]...Xen offers VGA PassThrough feature for ATI Radeon. > > Here is a link to a Youtube video to run Crysis 3 on a domU Windows 7 64 Bits with HD 7970 > > http://www.youtube.com/watch?v=GTnchEG4YtI&feature=player_embeddedNice, I''ve seen others and that''s make me think of Xen as a possible solution. (abusé, le fond d''écran ;) )> As you can see > > - I can use PCI Passthrough to add a XBOX 360 Paddle while playing Crysis 3 (see at 09:12 in the Video) > - I have sound too :)Can you share USB between different VMs? I mean, usb port 1 to VM #1, usb port 2 and 3 to VM #2...> But it works on Linux domU too. Here is another Youtube video link > > http://www.youtube.com/watch?feature=player_embedded&v=KzqOIMaBgX0 > > But you need to apply a few patches to able this feature with Xen 4.2 and over. Not very complicated to do. > Here are my instructions to do it but I know that there are currently better way to test it > > http://www.davidgis.fr/blog/index.php?2013/03/13/935-xen-43-vga-passthrough-ati-card-hd-7970-changeset-26706Thanks. I''ll take a look if I find a box do test.> [Extracted]...memory allocation > [Comment]...Xen - limited to 3GB for RAM. A few developers are working on overpassing this limitation.Oh my... Xen is dead to me. Seriously, I have 24 GB RAM, and use a big chunk of it, on my Mac active memory alone ranges from 6 to 11 GB, the remaining part is shared between inactive, and free.> [Extracted]...no snaphost possible > [Comment]...If you used Xen with LVM is then snahshotting is easier to manage ;)It would require either to snapshot every single VM, or to create a dedicated volume for each VM, not exactly what you get with ESX (snapshot per-VM independent of storage - even over NFS, including or excluding RAM). But well, a VM with passthrough in ESX as no snapshot capabilities, so relying on host FS is not a bad idea.> [Extracted] no sleep > [Comment] works on Xen for both Linux and Windows. The main issue is that you can reboot a domU without restarting the dom0.You''ve lost me here.> [Extracted] Is it possible to create a virtualized desktop with VGA and PCI passthrough for 2 or 3 VM''s running simultaneously (ie. each one with its own video card)? > [Comment] Yes it is doable. Never tested but from experience share with other Xen users I know that it should be run fine. > > Hope it helps.It does, but I''ve to admit the RAM limit is a no-go for me (for now, I''ll check facts in few months and follow the evolution on this topic). Thanks, Patrick
Eric Shelton
2013-Apr-02 23:26 UTC
Re: PCI/VGA passthrough: differences between Xen and ESXi?
David: What are the circumstances in which your comment about "Xen - limited to 3GB for RAM" applies? Is it in the case in which one uses multiple GPU cards? There are a number of HVM tutorial examples with well over 3GB: http://wiki.xen.org/wiki/Secondary_GPU_Passthrough (6GB Win7 HVM with passthrough of ATI 6970 and USB controller) http://wiki.xen.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial (6GB Win7 HVM with passthrough of ATI 68XX and multiple USB controllers) None of these tutorials mention a 3GB limitation. The only recent reference to a 3GB HVM issue during passthrough is here: http://comments.gmane.org/gmane.comp.emulators.xen.user/77214 which appears to link the issue with use of nvidia patches from here: http://www.davidgis.fr/download/xen-4.2_rev25240_gfx-passthrough-patchs.tar.bz2 From what I gather, nvidia cards perform a bit of undocumented voodoo (by design) that has not been fully accounted for. Elsewhere: http://new-wiki.xen.org/old-wiki/xenwiki/XenPCIpassthrough.html it sounds like there was a PV domain issue arising from e820 for which a reliable workaround was incorporated into the mainline code some time ago. Patrick: From the above examples, it seems like >3GB may be fine if you use ATI/AMD GPUs. I have had fairly good experiences with passthrough of a single AMD 6570 with the patch mentioned here: http://lists.xen.org/archives/html/xen-users/2013-02/msg00410.html although I did not explore how much or how little memory I could assign to it. Note, however, that patch may make certain assumptions, such as that only one GPU is being passed through. I''m not sure there is any unpatched release version (for example, 4.2.1) that reliably does GPU passthrough (PCI passthrough of less wonky devices has worked for some time). I think the proposed multi-GPU setup may be putting you out on the bleeding edge (on the other hand, it sounds like it is absolutely no available under ESXi). It''s probably not impossible. Maybe someone has already done it. However, how interested and/or comfortable are you with code & configuration tweaking, debug, and experimentation? - Eric>> Patrick, >> >> 1) Xen''s vocabulary >> >> dom0 = the main (first) hypervisor, hosting all virtual machines -- like your Mac OS X >> domU = any virtual machine. >> >> This is not the real meaning but here I try to stay "understandable". But for better understanding please refer to http://en.wikipedia.org/wiki/Xen >> >> My understanding is that your dom0 has to be replaced by Linux so you can run Xen. But for this part I am not a Mac OS X expert. >> >> 2) More info - details listed beow are only available for a Xen dom0 running on Linux. >> >> [Extracted]:....I''ve ended my testing with ESXi, because I was not able to passthrough sound device, and USB.... >> [Comment]: With Xen you can :) >> >> [Extracted]: ...with stock ATI Radeon..; >> [Comment]...Xen offers VGA PassThrough feature for ATI Radeon. >> >> Here is a link to a Youtube video to run Crysis 3 on a domU Windows 7 64 Bits with HD 7970 >> >> http://www.youtube.com/watch?v=GTnchEG4YtI&feature=player_embedded >> >> As you can see >> >> - I can use PCI Passthrough to add a XBOX 360 Paddle while playing Crysis 3 (see at 09:12 in the Video) >> >> - I have sound too :) >> >> But it works on Linux domU too. Here is another Youtube video link >> >> http://www.youtube.com/watch?feature=player_embedded&v=KzqOIMaBgX0 >> >> But you need to apply a few patches to able this feature with Xen 4.2 and over. Not very complicated to do. >> Here are my instructions to do it but I know that there are currently better way to test it >> >> http://www.davidgis.fr/blog/index.php?2013/03/13/935-xen-43-vga-passthrough-ati-card-hd-7970-changeset-26706 >> >> [Extracted]...memory allocation >> [Comment]...Xen - limited to 3GB for RAM. A few developers are working on overpassing this limitation. >> >> [Extracted]...no snaphost possible >> [Comment]...If you used Xen with LVM is then snahshotting is easier to manage ;) >> >> [Extracted] no sleep >> [Comment] works on Xen for both Linux and Windows. The main issue is that you can reboot a domU without restarting the dom0. >> >> >> [Extracted] Is it possible to create a virtualized desktop with VGA and PCI passthrough for 2 or 3 VM''s running simultaneously (ie. each one with its own video card)? >> [Comment] Yes it is doable. Never tested but from experience share with other Xen users I know that it should be run fine. >> >> Hope it helps. >> >> >> Kind regards. >> >> David.
David TECHER
2013-Apr-03 08:45 UTC
Re: PCI/VGA passthrough: differences between Xen and ESXi?
Thanks for clarifying. I am applying the original patches for nvidia. That''s the reason why I''ve got the 3GM limitation. Question 1: In your link http://lists.xen.org/archives/html/xen-users/2013-02/msg00410.html Did you apply the following patch ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch on Xen 4.3 unstable or Xen 4.2 stable? Question 2: Do you use this script ftp://ftp.enjellic.com/pub/xen/run-passthrough to start/stop your domU Question 3 Can you show a domU config file? Thanks for letting me know. ________________________________ De : Eric Shelton <eshelton@pobox.com> À : "xen-users@lists.xen.org" <xen-users@lists.xen.org>; Patrick Proniewski <patpro@patpro.net>; davidtecher@yahoo.fr Envoyé le : Mercredi 3 avril 2013 1h26 Objet : Re: [Xen-users] PCI/VGA passthrough: differences between Xen and ESXi? David: What are the circumstances in which your comment about "Xen - limited to 3GB for RAM" applies? Is it in the case in which one uses multiple GPU cards? There are a number of HVM tutorial examples with well over 3GB: http://wiki.xen.org/wiki/Secondary_GPU_Passthrough (6GB Win7 HVM with passthrough of ATI 6970 and USB controller) http://wiki.xen.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial (6GB Win7 HVM with passthrough of ATI 68XX and multiple USB controllers) None of these tutorials mention a 3GB limitation. The only recent reference to a 3GB HVM issue during passthrough is here: http://comments.gmane.org/gmane.comp.emulators.xen.user/77214 which appears to link the issue with use of nvidia patches from here: http://www.davidgis.fr/download/xen-4.2_rev25240_gfx-passthrough-patchs.tar.bz2 From what I gather, nvidia cards perform a bit of undocumented voodoo (by design) that has not been fully accounted for. Elsewhere: http://new-wiki.xen.org/old-wiki/xenwiki/XenPCIpassthrough.html it sounds like there was a PV domain issue arising from e820 for which a reliable workaround was incorporated into the mainline code some time ago. Patrick: From the above examples, it seems like >3GB may be fine if you use ATI/AMD GPUs. I have had fairly good experiences with passthrough of a single AMD 6570 with the patch mentioned here: http://lists.xen.org/archives/html/xen-users/2013-02/msg00410.html although I did not explore how much or how little memory I could assign to it. Note, however, that patch may make certain assumptions, such as that only one GPU is being passed through. I''m not sure there is any unpatched release version (for example, 4.2.1) that reliably does GPU passthrough (PCI passthrough of less wonky devices has worked for some time). I think the proposed multi-GPU setup may be putting you out on the bleeding edge (on the other hand, it sounds like it is absolutely no available under ESXi). It''s probably not impossible. Maybe someone has already done it. However, how interested and/or comfortable are you with code & configuration tweaking, debug, and experimentation? - Eric>> Patrick, >> >> 1) Xen''s vocabulary >> >> dom0 = the main (first) hypervisor, hosting all virtual machines -- like your Mac OS X >> domU = any virtual machine. >> >> This is not the real meaning but here I try to stay "understandable". But for better understanding please refer to http://en.wikipedia.org/wiki/Xen >> >> My understanding is that your dom0 has to be replaced by Linux so you can run Xen. But for this part I am not a Mac OS X expert. >> >> 2) More info - details listed beow are only available for a Xen dom0 running on Linux. >> >> [Extracted]:....I''ve ended my testing with ESXi, because I was not able to passthrough sound device, and USB.... >> [Comment]: With Xen you can :) >> >> [Extracted]: ...with stock ATI Radeon..; >> [Comment]...Xen offers VGA PassThrough feature for ATI Radeon. >> >> Here is a link to a Youtube video to run Crysis 3 on a domU Windows 7 64 Bits with HD 7970 >> >> http://www.youtube.com/watch?v=GTnchEG4YtI&feature=player_embedded >> >> As you can see >> >> - I can use PCI Passthrough to add a XBOX 360 Paddle while playing Crysis 3 (see at 09:12 in the Video) >> >> - I have sound too :) >> >> But it works on Linux domU too. Here is another Youtube video link >> >> http://www.youtube.com/watch?feature=player_embedded&v=KzqOIMaBgX0 >> >> But you need to apply a few patches to able this feature with Xen 4.2 and over. Not very complicated to do. >> Here are my instructions to do it but I know that there are currently better way to test it >> >> http://www.davidgis.fr/blog/index.php?2013/03/13/935-xen-43-vga-passthrough-ati-card-hd-7970-changeset-26706 >> >> [Extracted]...memory allocation >> [Comment]...Xen - limited to 3GB for RAM. A few developers are working on overpassing this limitation. >> >> [Extracted]...no snaphost possible >> [Comment]...If you used Xen with LVM is then snahshotting is easier to manage ;) >> >> [Extracted] no sleep >> [Comment] works on Xen for both Linux and Windows. The main issue is that you can reboot a domU without restarting the dom0. >> >> >> [Extracted] Is it possible to create a virtualized desktop with VGA and PCI passthrough for 2 or 3 VM''s running simultaneously (ie. each one with its own video card)? >> [Comment] Yes it is doable. Never tested but from experience share with other Xen users I know that it should be run fine. >> >> Hope it helps. >> >> >> Kind regards. >> >> David._______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
David TECHER
2013-Apr-03 09:06 UTC
Re: PCI/VGA passthrough: differences between Xen and ESXi?
My understanding is that there are currently two types of patches. Which works for Xen 4.3? To be more precised I would like to test it for Xen 4.3 Do you mind to provide me all technical details? (make, configue, patch < ????) Got a HD 7970. I would like to test it with RAM 3 GB. Thanks. ________________________________ De : David TECHER <davidtecher@yahoo.fr> À : Eric Shelton <eshelton@pobox.com>; "xen-users@lists.xen.org" <xen-users@lists.xen.org>; Patrick Proniewski <patpro@patpro.net> Envoyé le : Mercredi 3 avril 2013 10h45 Objet : Re: [Xen-users] PCI/VGA passthrough: differences between Xen and ESXi? Thanks for clarifying. I am applying the original patches for nvidia. That''s the reason why I''ve got the 3GM limitation. Question 1: In your link http://lists.xen.org/archives/html/xen-users/2013-02/msg00410.html Did you apply the following patch ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch on Xen 4.3 unstable or Xen 4.2 stable? Question 2: Do you use this script ftp://ftp.enjellic.com/pub/xen/run-passthrough to start/stop your domU Question 3 Can you show a domU config file? Thanks for letting me know. ________________________________ De : Eric Shelton <eshelton@pobox.com> À : "xen-users@lists.xen.org" <xen-users@lists.xen.org>; Patrick Proniewski <patpro@patpro.net>; davidtecher@yahoo.fr Envoyé le : Mercredi 3 avril 2013 1h26 Objet : Re: [Xen-users] PCI/VGA passthrough: differences between Xen and ESXi? David: What are the circumstances in which your comment about "Xen - limited to 3GB for RAM" applies? Is it in the case in which one uses multiple GPU cards? There are a number of HVM tutorial examples with well over 3GB: http://wiki.xen.org/wiki/Secondary_GPU_Passthrough (6GB Win7 HVM with passthrough of ATI 6970 and USB controller) http://wiki.xen.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial (6GB Win7 HVM with passthrough of ATI 68XX and multiple USB controllers) None of these tutorials mention a 3GB limitation. The only recent reference to a 3GB HVM issue during passthrough is here: http://comments.gmane.org/gmane.comp.emulators.xen.user/77214 which appears to link the issue with use of nvidia patches from here: http://www.davidgis.fr/download/xen-4.2_rev25240_gfx-passthrough-patchs.tar.bz2 From what I gather, nvidia cards perform a bit of undocumented voodoo (by design) that has not been fully accounted for. Elsewhere: http://new-wiki.xen.org/old-wiki/xenwiki/XenPCIpassthrough.html it sounds like there was a PV domain issue arising from e820 for which a reliable workaround was incorporated into the mainline code some time ago. Patrick: From the above examples, it seems like >3GB may be fine if you use ATI/AMD GPUs. I have had fairly good experiences with passthrough of a single AMD 6570 with the patch mentioned here: http://lists.xen.org/archives/html/xen-users/2013-02/msg00410.html although I did not explore how much or how little memory I could assign to it. Note, however, that patch may make certain assumptions, such as that only one GPU is being passed through. I''m not sure there is any unpatched release version (for example, 4.2.1) that reliably does GPU passthrough (PCI passthrough of less wonky devices has worked for some time). I think the proposed multi-GPU setup may be putting you out on the bleeding edge (on the other hand, it sounds like it is absolutely no available under ESXi). It''s probably not impossible. Maybe someone has already done it. However, how interested and/or comfortable are you with code & configuration tweaking, debug, and experimentation? - Eric>> Patrick, >> >> 1) Xen''s vocabulary >> >> dom0 = the main (first) hypervisor, hosting all virtual machines -- like your Mac OS X >> domU= any virtual machine.>> >> This is not the real meaning but here I try to stay "understandable". But for better understanding please refer to http://en.wikipedia.org/wiki/Xen >> >> My understanding is that your dom0 has to be replaced by Linux so you can run Xen. But for this part I am not a Mac OS X expert. >> >> 2) More info - details listed beow are only available for a Xen dom0 running on Linux. >> >> [Extracted]:....I''ve ended my testing with ESXi, because I was not able to passthrough sound device, and USB.... >> [Comment]: With Xen you can :) >> >> [Extracted]: ...with stock ATI Radeon..; >> [Comment]...Xen offers VGA PassThrough feature for ATI Radeon. >> >> Here is a link to a Youtube video to run Crysis 3 on a domU Windows 7 64 Bits with HD7970>> >> http://www.youtube.com/watch?v=GTnchEG4YtI&feature=player_embedded >> >> As you can see >> >> - I can use PCI Passthrough to add a XBOX 360 Paddle while playing Crysis 3 (see at 09:12 in the Video) >> >> - I have sound too :) >> >> But it works on Linux domU too. Here is another Youtube video link >> >> http://www.youtube.com/watch?feature=player_embedded&v=KzqOIMaBgX0 >> >> But you need to apply a few patches to able this feature with Xen 4.2 and over. Not very complicated to do. >> Here are my instructions to do it but I know that there are currently better way to test it >> >> http://www.davidgis.fr/blog/index.php?2013/03/13/935-xen-43-vga-passthrough-ati-card-hd-7970-changeset-26706 >> >> [Extracted]...memory allocation >> [Comment]...Xen - limited to 3GB for RAM. A few developers are working on overpassing this limitation. >> >> [Extracted]...no snaphost possible >> [Comment]...If you used Xen with LVM is then snahshotting is easier to manage ;) >> >> [Extracted] no sleep >> [Comment] works on Xen for both Linux and Windows. The main issue is that you can reboot a domU without restarting the dom0. >> >> >> [Extracted] Is it possible to create a virtualized desktop with VGA and PCI passthrough for 2 or 3 VM''s running simultaneously (ie. each one with its own video card)? >>[Comment] Yes it is doable. Never tested but from experience share with other Xen users I know that it should be run fine.>> >> Hope it helps. >> >> >> Kind regards. >> >> David._______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Eric Shelton
2013-Apr-03 16:01 UTC
Re: PCI/VGA passthrough: differences between Xen and ESXi?
1) I applied the patch to 4.3 unstable (back in February). I recall it applying cleanly (thanks to Dr. Wettstein''s work in updating the original patches). However, there is a build issue when the patched files are used to build minios, as its libraries do not provide iopl(). On my initial pass, I may have #ifdef-ed out the iopl() calls during minios build time, on the presumption that this code path is not exercised by minios. However, not being fully confident in that presumption, in some later work I identified a more appropriate mechanism: an existing hypervisor call for iopl. I think revised code was written, but remains untested. 2) I am using what I saw as the core aspects of that script: unbind_devices(), re-enabling the devices, and the use of vbetool to reinitialize the display (which returns the text console in my case). Although I am doing passthrough of a USB3 controller as well, I did not bother with rebinding the USB driver, as I simply surrendered use of those ports for Windows. I also am not waiting for the VM to exit; instead, I split it the single script into before and after scripts that were manually invoked. I think the runtime unbind, bind, and vbetool, versus giving the PCI devices over to pciback at boot time, makes a difference in terms of whether you can stop & restart the passthrough VM. With this config, it is no problem for me. I heard many reports of people starting up the Windows VM once, but not again without a reboot. 3+) Tonight I can pull together what I am using and post it up here. Very little modification of Dr. Wettstein''s patch was done. However, at this stage, it seems to be the little things that make the difference between GPU passthrough working or not. My understanding is that there are many intentional quirks in GPUs to be in compliance with certain HDCP requirements. Unfortunately, I think the universe of possible secret register access sequences, etc. will make it impossible, even with 1:1 mapping in addition, to 100% convince the display driver, which would be needed to get to virtualized HDCP compliance (I am interested in a virtualized HTPC environment). It sounds like, from what I have heard of your work, that NVidia''s quirks are especially difficult to deal with (not necessarily intentionally). It was enough to make me abandon by existing NVidia GPU in favor of an AMD solution. Also, as I think I noted in an earlier post, one of the qemu developers has, perhaps independently, implemented the necessary quirks for AMD cards. His code effects pretty much the same things as the patch I used. I think there is at least a patch against mainline qemu in qemu-devel, if it has not been incorporated already. He has also been doing a fair amount of PCIe work, which I think may be important for NVidia, assuming their use of extended config space is causing some of the difficulties. It would be nice to see enough code get into Xen 4.3 to get AMD cards working, but with the switch away from qemu-traditional, which is what the patch is for, this may now be more reliant on the work already being done in qemu-devel being mainlined into qemu. - Eric On Wed, Apr 3, 2013 at 4:45 AM, David TECHER <davidtecher@yahoo.fr> wrote:> Thanks for clarifying. > > I am applying the original patches for nvidia. That''s the reason why I''ve > got the 3GM limitation. > > Question 1: > > In your link > > http://lists.xen.org/archives/html/xen-users/2013-02/msg00410.html > > Did you apply the following patch > > ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch > > on Xen 4.3 unstable or Xen 4.2 stable? > > Question 2: > > Do you use this script > > ftp://ftp.enjellic.com/pub/xen/run-passthrough > > to start/stop your domU > > Question 3 > > Can you show a domU config file? > > Thanks for letting me know. > > > > ________________________________ > De : Eric Shelton <eshelton@pobox.com> > À : "xen-users@lists.xen.org" <xen-users@lists.xen.org>; Patrick Proniewski > <patpro@patpro.net>; davidtecher@yahoo.fr > Envoyé le : Mercredi 3 avril 2013 1h26 > Objet : Re: [Xen-users] PCI/VGA passthrough: differences between Xen and > ESXi? > > David: > What are the circumstances in which your comment about "Xen - limited > to 3GB for RAM" applies? Is it in the case in which one uses multiple > GPU cards? > > There are a number of HVM tutorial examples with well over 3GB: > http://wiki.xen.org/wiki/Secondary_GPU_Passthrough (6GB Win7 HVM with > passthrough of ATI 6970 and USB controller) > http://wiki.xen.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial > (6GB Win7 HVM with passthrough of ATI 68XX and multiple USB > controllers) > None of these tutorials mention a 3GB limitation. > > The only recent reference to a 3GB HVM issue during passthrough is here: > http://comments.gmane.org/gmane.comp.emulators.xen.user/77214 > which appears to link the issue with use of nvidia patches from here: > http://www.davidgis.fr/download/xen-4.2_rev25240_gfx-passthrough-patchs.tar.bz2 > From what I gather, nvidia cards perform a bit of undocumented voodoo > (by design) that has not been fully accounted for. > > Elsewhere: > http://new-wiki.xen.org/old-wiki/xenwiki/XenPCIpassthrough.html > it sounds like there was a PV domain issue arising from e820 for which > a reliable workaround was incorporated into the mainline code some > time ago. > > > Patrick: > From the above examples, it seems like >3GB may be fine if you use > ATI/AMD GPUs. I have had fairly good experiences with passthrough of > a single AMD 6570 with the patch mentioned here: > http://lists.xen.org/archives/html/xen-users/2013-02/msg00410.html > although I did not explore how much or how little memory I could > assign to it. Note, however, that patch may make certain assumptions, > such as that only one GPU is being passed through. I''m not sure there > is any unpatched release version (for example, 4.2.1) that reliably > does GPU passthrough (PCI passthrough of less wonky devices has worked > for some time). > > I think the proposed multi-GPU setup may be putting you out on the > bleeding edge (on the other hand, it sounds like it is absolutely no > available under ESXi). It''s probably not impossible. Maybe someone > has already done it. However, how interested and/or comfortable are > you with code & configuration tweaking, debug, and experimentation? > > - Eric > >>> Patrick, >>> >>> 1) Xen''s vocabulary >>> >>> dom0 = the main (first) hypervisor, hosting all virtual machines -- like >>> your Mac OS X >>> domU = any virtual machine. >>> >>> This is not the real meaning but here I try to stay "understandable". But >>> for better understanding please refer to http://en.wikipedia.org/wiki/Xen >>> >>> My understanding is that your dom0 has to be replaced by Linux so you can >>> run Xen. But for this part I am not a Mac OS X expert. >>> >>> 2) More info - details listed beow are only available for a Xen dom0 >>> running on Linux. >>> >>> [Extracted]:....I''ve ended my testing with ESXi, because I was not able >>> to passthrough sound device, and USB.... >>> [Comment]: With Xen you can :) >>> >>> [Extracted]: ...with stock ATI Radeon..; >>> [Comment]...Xen offers VGA PassThrough feature for ATI Radeon. >>> >>> Here is a link to a Youtube video to run Crysis 3 on a domU Windows 7 64 >>> Bits with HD 7970 >>> >>> http://www.youtube.com/watch?v=GTnchEG4YtI&feature=player_embedded >>> >>> As you can see >>> >>> - I can use PCI Passthrough to add a XBOX 360 Paddle while playing Crysis >>> 3 (see at 09:12 in the Video) >>> >>> - I have sound too :) >>> >>> But it works on Linux domU too. Here is another Youtube video link >>> >>> http://www.youtube.com/watch?feature=player_embedded&v=KzqOIMaBgX0 >>> >>> But you need to apply a few patches to able this feature with Xen 4.2 and >>> over. Not very complicated to do. >>> Here are my instructions to do it but I know that there are currently >>> better way to test it >>> >>> >>> http://www.davidgis.fr/blog/index.php?2013/03/13/935-xen-43-vga-passthrough-ati-card-hd-7970-changeset-26706 >>> >>> [Extracted]...memory allocation >>> [Comment]...Xen - limited to 3GB for RAM. A few developers are working on >>> overpassing this limitation. >>> >>> [Extracted]...no snaphost possible >>> [Comment]...If you used Xen with LVM is then snahshotting is easier to >>> manage ;) >>> >>> [Extracted] no sleep >>> [Comment] works on Xen for both Linux and Windows. The main issue is that >>> you can reboot a domU without restarting the dom0. >>> >>> >>> [Extracted] Is it possible to create a virtualized desktop with VGA and >>> PCI passthrough for 2 or 3 VM''s running simultaneously (ie. each one with >>> its own video card)? >>> [Comment] Yes it is doable. Never tested but from experience share with >>> other Xen users I know that it should be run fine. >>> >>> Hope it helps. >>> >>> >>> Kind regards. >>> >>> David. > >
jocelyn falempe
2013-Apr-03 19:39 UTC
Re: PCI/VGA passthrough: differences between Xen and ESXi?
if you have 2 graphics cards, I would suggest to use PCI passthrough instead of VGA passthrough. no patches needed, you can start/stop the VM when you want. you can even play 3D game at the same time on the Dom0 and the DomU. you can find more details on the setup on my blog : http://kdj0c.wordpress.com/2013/02/12/xen-pci-passthrough/ and another source : http://gro.solexiv.de/2012/08/pci-passthrough-howto/ -- Jocelyn On 04/03/2013 06:01 PM, Eric Shelton wrote:> 1) I applied the patch to 4.3 unstable (back in February). I recall > it applying cleanly (thanks to Dr. Wettstein''s work in updating the > original patches). However, there is a build issue when the patched > files are used to build minios, as its libraries do not provide > iopl(). On my initial pass, I may have #ifdef-ed out the iopl() calls > during minios build time, on the presumption that this code path is > not exercised by minios. However, not being fully confident in that > presumption, in some later work I identified a more appropriate > mechanism: an existing hypervisor call for iopl. I think revised code > was written, but remains untested. > > 2) I am using what I saw as the core aspects of that script: > unbind_devices(), re-enabling the devices, and the use of vbetool to > reinitialize the display (which returns the text console in my case). > Although I am doing passthrough of a USB3 controller as well, I did > not bother with rebinding the USB driver, as I simply surrendered use > of those ports for Windows. I also am not waiting for the VM to exit; > instead, I split it the single script into before and after scripts > that were manually invoked. > > I think the runtime unbind, bind, and vbetool, versus giving the PCI > devices over to pciback at boot time, makes a difference in terms of > whether you can stop & restart the passthrough VM. With this config, > it is no problem for me. I heard many reports of people starting up > the Windows VM once, but not again without a reboot. > > 3+) Tonight I can pull together what I am using and post it up here. > Very little modification of Dr. Wettstein''s patch was done. However, > at this stage, it seems to be the little things that make the > difference between GPU passthrough working or not. > > My understanding is that there are many intentional quirks in GPUs to > be in compliance with certain HDCP requirements. Unfortunately, I > think the universe of possible secret register access sequences, etc. > will make it impossible, even with 1:1 mapping in addition, to 100% > convince the display driver, which would be needed to get to > virtualized HDCP compliance (I am interested in a virtualized HTPC > environment). It sounds like, from what I have heard of your work, > that NVidia''s quirks are especially difficult to deal with (not > necessarily intentionally). It was enough to make me abandon by > existing NVidia GPU in favor of an AMD solution. > > Also, as I think I noted in an earlier post, one of the qemu > developers has, perhaps independently, implemented the necessary > quirks for AMD cards. His code effects pretty much the same things as > the patch I used. I think there is at least a patch against mainline > qemu in qemu-devel, if it has not been incorporated already. He has > also been doing a fair amount of PCIe work, which I think may be > important for NVidia, assuming their use of extended config space is > causing some of the difficulties. > > It would be nice to see enough code get into Xen 4.3 to get AMD cards > working, but with the switch away from qemu-traditional, which is what > the patch is for, this may now be more reliant on the work already > being done in qemu-devel being mainlined into qemu. > > - Eric
David TECHER
2013-Apr-03 19:49 UTC
Re: PCI/VGA passthrough: differences between Xen and ESXi?
Thanks for this wonderful story but I was asking 1) for your revision number/changeset for Xen 4.3 (xl info should return something) hg clone -r ???? ./configure make ??? patch -p1 <???? make ??? 2) any domU configuration file ? ________________________________ De : Eric Shelton <eshelton@pobox.com> À : David TECHER <davidtecher@yahoo.fr> Cc : "xen-users@lists.xen.org" <xen-users@lists.xen.org>; Patrick Proniewski <patpro@patpro.net> Envoyé le : Mercredi 3 avril 2013 18h01 Objet : Re: [Xen-users] PCI/VGA passthrough: differences between Xen and ESXi? 1) I applied the patch to 4.3 unstable (back in February). I recall it applying cleanly (thanks to Dr. Wettstein''s work in updating the original patches). However, there is a build issue when the patched files are used to build minios, as its libraries do not provide iopl(). On my initial pass, I may have #ifdef-ed out the iopl() calls during minios build time, on the presumption that this code path is not exercised by minios. However, not being fully confident in that presumption, in some later work I identified a more appropriate mechanism: an existing hypervisor call for iopl. I think revised code was written, but remains untested. 2) I am using what I saw as the core aspects of that script: unbind_devices(), re-enabling the devices, and the use of vbetool to reinitialize the display (which returns the text console in my case). Although I am doing passthrough of a USB3 controller as well, I did not bother with rebinding the USB driver, as I simply surrendered use of those ports for Windows. I also am not waiting for the VM to exit; instead, I split it the single script into before and after scripts that were manually invoked. I think the runtime unbind, bind, and vbetool, versus giving the PCI devices over to pciback at boot time, makes a difference in terms of whether you can stop & restart the passthrough VM. With this config, it is no problem for me. I heard many reports of people starting up the Windows VM once, but not again without a reboot. 3+) Tonight I can pull together what I am using and post it up here. Very little modification of Dr. Wettstein''s patch was done. However, at this stage, it seems to be the little things that make the difference between GPU passthrough working or not. My understanding is that there are many intentional quirks in GPUs to be in compliance with certain HDCP requirements. Unfortunately, I think the universe of possible secret register access sequences, etc. will make it impossible, even with 1:1 mapping in addition, to 100% convince the display driver, which would be needed to get to virtualized HDCP compliance (I am interested in a virtualized HTPC environment). It sounds like, from what I have heard of your work, that NVidia''s quirks are especially difficult to deal with (not necessarily intentionally). It was enough to make me abandon by existing NVidia GPU in favor of an AMD solution. Also, as I think I noted in an earlier post, one of the qemu developers has, perhaps independently, implemented the necessary quirks for AMD cards. His code effects pretty much the same things as the patch I used. I think there is at least a patch against mainline qemu in qemu-devel, if it has not been incorporated already. He has also been doing a fair amount of PCIe work, which I think may be important for NVidia, assuming their use of extended config space is causing some of the difficulties. It would be nice to see enough code get into Xen 4.3 to get AMD cards working, but with the switch away from qemu-traditional, which is what the patch is for, this may now be more reliant on the work already being done in qemu-devel being mainlined into qemu. - Eric On Wed, Apr 3, 2013 at 4:45 AM, David TECHER <davidtecher@yahoo.fr> wrote:> Thanks for clarifying. > > I am applying the original patches for nvidia. That''s the reason why I''ve > got the 3GM limitation. > > Question 1: > > In your link > > http://lists.xen.org/archives/html/xen-users/2013-02/msg00410.html > > Did you apply the following patch > > ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch > > on Xen 4.3 unstable or Xen 4.2 stable? > > Question 2: > > Do you use this script > > ftp://ftp.enjellic.com/pub/xen/run-passthrough > > to start/stop your domU > > Question 3 > > Can you show a domU config file? > > Thanks for letting me know. > > > > ________________________________ > De : Eric Shelton <eshelton@pobox.com> > À : "xen-users@lists.xen.org" <xen-users@lists.xen.org>; Patrick Proniewski > <patpro@patpro.net>; davidtecher@yahoo.fr > Envoyé le : Mercredi 3 avril 2013 1h26 > Objet : Re: [Xen-users] PCI/VGA passthrough: differences between Xen and > ESXi? > > David: > What are the circumstances in which your comment about "Xen - limited > to 3GB for RAM" applies? Is it in the case in which one uses multiple > GPU cards? > > There are a number of HVM tutorial examples with well over 3GB: > http://wiki.xen.org/wiki/Secondary_GPU_Passthrough (6GB Win7 HVM with > passthrough of ATI 6970 and USB controller) > http://wiki.xen.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial > (6GB Win7 HVM with passthrough of ATI 68XX and multiple USB > controllers) > None of these tutorials mention a 3GB limitation. > > The only recent reference to a 3GB HVM issue during passthrough is here: > http://comments.gmane.org/gmane.comp.emulators.xen.user/77214 > which appears to link the issue with use of nvidia patches from here: > http://www.davidgis.fr/download/xen-4.2_rev25240_gfx-passthrough-patchs.tar.bz2 > From what I gather, nvidia cards perform a bit of undocumented voodoo > (by design) that has not been fully accounted for. > > Elsewhere: > http://new-wiki.xen.org/old-wiki/xenwiki/XenPCIpassthrough.html > it sounds like there was a PV domain issue arising from e820 for which > a reliable workaround was incorporated into the mainline code some > time ago. > > > Patrick: > From the above examples, it seems like >3GB may be fine if you use > ATI/AMD GPUs. I have had fairly good experiences with passthrough of > a single AMD 6570 with the patch mentioned here: > http://lists.xen.org/archives/html/xen-users/2013-02/msg00410.html > although I did not explore how much or how little memory I could > assign to it. Note, however, that patch may make certain assumptions, > such as that only one GPU is being passed through. I''m not sure there > is any unpatched release version (for example, 4.2.1) that reliably > does GPU passthrough (PCI passthrough of less wonky devices has worked > for some time). > > I think the proposed multi-GPU setup may be putting you out on the > bleeding edge (on the other hand, it sounds like it is absolutely no > available under ESXi). It''s probably not impossible. Maybe someone > has already done it. However, how interested and/or comfortable are > you with code & configuration tweaking, debug, and experimentation? > > - Eric > >>> Patrick, >>> >>> 1) Xen''s vocabulary >>> >>> dom0 = the main (first) hypervisor, hosting all virtual machines -- like >>> your Mac OS X >>> domU = any virtual machine. >>> >>> This is not the real meaning but here I try to stay "understandable". But >>> for better understanding please refer to http://en.wikipedia.org/wiki/Xen >>> >>> My understanding is that your dom0 has to be replaced by Linux so you can >>> run Xen. But for this part I am not a Mac OS X expert. >>> >>> 2) More info - details listed beow are only available for a Xen dom0 >>> running on Linux. >>> >>> [Extracted]:....I''ve ended my testing with ESXi, because I was not able >>> to passthrough sound device, and USB.... >>> [Comment]: With Xen you can :) >>> >>> [Extracted]: ...with stock ATI Radeon..; >>> [Comment]...Xen offers VGA PassThrough feature for ATI Radeon. >>> >>> Here is a link to a Youtube video to run Crysis 3 on a domU Windows 7 64 >>> Bits with HD 7970 >>> >>> http://www.youtube.com/watch?v=GTnchEG4YtI&feature=player_embedded >>> >>> As you can see >>> >>> - I can use PCI Passthrough to add a XBOX 360 Paddle while playing Crysis >>> 3 (see at 09:12 in the Video) >>> >>> - I have sound too :) >>> >>> But it works on Linux domU too. Here is another Youtube video link >>> >>> http://www.youtube.com/watch?feature=player_embedded&v=KzqOIMaBgX0 >>> >>> But you need to apply a few patches to able this feature with Xen 4.2 and >>> over. Not very complicated to do. >>> Here are my instructions to do it but I know that there are currently >>> better way to test it >>> >>> >>> http://www.davidgis.fr/blog/index.php?2013/03/13/935-xen-43-vga-passthrough-ati-card-hd-7970-changeset-26706 >>> >>> [Extracted]...memory allocation >>> [Comment]...Xen - limited to 3GB for RAM. A few developers are working on >>> overpassing this limitation. >>> >>> [Extracted]...no snaphost possible >>> [Comment]...If you used Xen with LVM is then snahshotting is easier to >>> manage ;) >>> >>> [Extracted] no sleep >>> [Comment] works on Xen for both Linux and Windows. The main issue is that >>> you can reboot a domU without restarting the dom0. >>> >>> >>> [Extracted] Is it possible to create a virtualized desktop with VGA and >>> PCI passthrough for 2 or 3 VM''s running simultaneously (ie. each one with >>> its own video card)? >>> [Comment] Yes it is doable. Never tested but from experience share with >>> other Xen users I know that it should be run fine. >>> >>> Hope it helps. >>> >>> >>> Kind regards. >>> >>> David. > >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Ricardo Jesus
2013-Apr-03 20:18 UTC
Re: PCI/VGA passthrough: differences between Xen and ESXi?
Same here. I have an AMD HD7850 powering a Windows 8 domU and an AMD HD5450 running a Debian Wheezy domU. The dom0 is Debian Wheezy with the distribution''s stock Xen 4.1.4, xm, pci_stub and no patches whatsoever. A while back I had a similar setup, this time with VGA Passthrough of primary display adapter Intel HD4000. At the time I blogged the HowTo at http://linux-bsd-sharing.blogspot.pt/2012/10/howto-xen-413-windows-8-hvm-domu-with.html. The same principles apply to my current setup. Best of luck, Ricardo Jesus. On Wed, Apr 3, 2013 at 8:39 PM, jocelyn falempe <jocelyn.falempe@free.fr> wrote:> if you have 2 graphics cards, I would suggest to use PCI passthrough instead > of VGA passthrough. > no patches needed, you can start/stop the VM when you want. you can even > play 3D game at the same time on the Dom0 and the DomU. > > you can find more details on the setup on my blog : > http://kdj0c.wordpress.com/2013/02/12/xen-pci-passthrough/ > > and another source : > http://gro.solexiv.de/2012/08/pci-passthrough-howto/ > > -- > > Jocelyn > > > On 04/03/2013 06:01 PM, Eric Shelton wrote: >> >> 1) I applied the patch to 4.3 unstable (back in February). I recall >> it applying cleanly (thanks to Dr. Wettstein''s work in updating the >> original patches). However, there is a build issue when the patched >> files are used to build minios, as its libraries do not provide >> iopl(). On my initial pass, I may have #ifdef-ed out the iopl() calls >> during minios build time, on the presumption that this code path is >> not exercised by minios. However, not being fully confident in that >> presumption, in some later work I identified a more appropriate >> mechanism: an existing hypervisor call for iopl. I think revised code >> was written, but remains untested. >> >> 2) I am using what I saw as the core aspects of that script: >> unbind_devices(), re-enabling the devices, and the use of vbetool to >> reinitialize the display (which returns the text console in my case). >> Although I am doing passthrough of a USB3 controller as well, I did >> not bother with rebinding the USB driver, as I simply surrendered use >> of those ports for Windows. I also am not waiting for the VM to exit; >> instead, I split it the single script into before and after scripts >> that were manually invoked. >> >> I think the runtime unbind, bind, and vbetool, versus giving the PCI >> devices over to pciback at boot time, makes a difference in terms of >> whether you can stop & restart the passthrough VM. With this config, >> it is no problem for me. I heard many reports of people starting up >> the Windows VM once, but not again without a reboot. >> >> 3+) Tonight I can pull together what I am using and post it up here. >> Very little modification of Dr. Wettstein''s patch was done. However, >> at this stage, it seems to be the little things that make the >> difference between GPU passthrough working or not. >> >> My understanding is that there are many intentional quirks in GPUs to >> be in compliance with certain HDCP requirements. Unfortunately, I >> think the universe of possible secret register access sequences, etc. >> will make it impossible, even with 1:1 mapping in addition, to 100% >> convince the display driver, which would be needed to get to >> virtualized HDCP compliance (I am interested in a virtualized HTPC >> environment). It sounds like, from what I have heard of your work, >> that NVidia''s quirks are especially difficult to deal with (not >> necessarily intentionally). It was enough to make me abandon by >> existing NVidia GPU in favor of an AMD solution. >> >> Also, as I think I noted in an earlier post, one of the qemu >> developers has, perhaps independently, implemented the necessary >> quirks for AMD cards. His code effects pretty much the same things as >> the patch I used. I think there is at least a patch against mainline >> qemu in qemu-devel, if it has not been incorporated already. He has >> also been doing a fair amount of PCIe work, which I think may be >> important for NVidia, assuming their use of extended config space is >> causing some of the difficulties. >> >> It would be nice to see enough code get into Xen 4.3 to get AMD cards >> working, but with the switch away from qemu-traditional, which is what >> the patch is for, this may now be more reliant on the work already >> being done in qemu-devel being mainlined into qemu. >> >> - Eric > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users
David TECHER
2013-Apr-04 01:00 UTC
Re: PCI/VGA passthrough: differences between Xen and ESXi?
I did it myself :)
Indeed this patch seems to be very interesting.
rev=26771
hg clone -r $rev http://xenbits.xensource.com/staging/xen-unstable.hg/
xen-unstable.hg-rev-${rev}
./configure
cd tools
make
make clean
cd ..
wget ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch -O - | sed
-e "s:qemu-xen-traditional:qemu-xen-traditional-dir-remote:g" | patch
-p1
make xen && make install-xen
make tools && make install-tools
Thanks for your help about iopl(). Agreed that iopl(3) and iopl(0) for stubdom
are a bit complicated. I commented out these functions so I can build stubdom
(shame on me!)
make stubdom
make install-stubdom
Using the script to download I was able (after testing/updating only the start
process)
- to start my domU (Linux Ubuntu 12.04 32). I was able to test it for RAM over
than 3GB :). 6GB and 6 vcpus. All my games with Wine works perfectly, VLC,
smplayer are OK :)
- to start my domU (Windows 7). I didn''t test it for RAM over than 3GB.
I will do a full test by reinstalling a Windows 7 64 Bits.
Thanks for helping.
________________________________
De : David TECHER <davidtecher@yahoo.fr>
À : Eric Shelton <eshelton@pobox.com>
Cc : "xen-users@lists.xen.org" <xen-users@lists.xen.org>;
Patrick Proniewski <patpro@patpro.net>
Envoyé le : Mercredi 3 avril 2013 21h49
Objet : Re: [Xen-users] PCI/VGA passthrough: differences between Xen and ESXi?
Thanks for this wonderful story but
I was asking
1) for your revision number/changeset for Xen 4.3 (xl info should return
something)
hg clone -r ????
./configure
make ???
patch -p1 <????
make ???
2) any domU configuration file ?
________________________________
De : Eric Shelton <eshelton@pobox.com>
À : David TECHER <davidtecher@yahoo.fr>
Cc : "xen-users@lists.xen.org" <xen-users@lists.xen.org>;
Patrick Proniewski <patpro@patpro.net>
Envoyé le : Mercredi 3 avril 2013 18h01
Objet : Re: [Xen-users] PCI/VGA passthrough: differences between Xen and ESXi?
1) I applied the patch to 4.3 unstable (back in February). I recall
it applying cleanly (thanks to Dr. Wettstein''s work in updating the
original patches). However, there is a build issue when the patched
files are used to build minios, as its libraries do not provide
iopl(). On my initial pass, I may have #ifdef-ed out the iopl() calls
during minios build time, on the presumption that this code path is
not exercised by minios. However, not being fully confident in that
presumption, in some later work I identified a more appropriate
mechanism: an
existing hypervisor call for iopl. I think revised code
was written, but remains untested.
2) I am using what I saw as the core aspects of that script:
unbind_devices(), re-enabling the devices, and the use of vbetool to
reinitialize the display (which returns the text console in my case).
Although I am doing passthrough of a USB3 controller as well, I did
not bother with rebinding the USB driver, as I simply surrendered use
of those ports for Windows. I also am not waiting for the VM to exit;
instead, I split it the single script into before and after scripts
that were manually invoked.
I think the runtime unbind, bind, and vbetool, versus giving the PCI
devices over to pciback at boot time, makes a difference in terms of
whether you can stop & restart the passthrough VM. With this config,
it is no problem for me. I heard many reports of people starting up
the Windows VM
once, but not again without a reboot.
3+) Tonight I can pull together what I am using and post it up here.
Very little modification of Dr. Wettstein''s patch was done. However,
at this stage, it seems to be the little things that make the
difference between GPU passthrough working or not.
My understanding is that there are many intentional quirks in GPUs to
be in compliance with certain HDCP requirements. Unfortunately, I
think the universe of possible secret register access sequences, etc.
will make it impossible, even with 1:1 mapping in addition, to 100%
convince the display driver, which would be needed to get to
virtualized HDCP compliance (I am interested in a virtualized HTPC
environment). It sounds like, from what I have heard of your work,
that NVidia''s quirks are especially difficult to deal with (not
necessarily intentionally). It was enough to make me abandon
by
existing NVidia GPU in favor of an AMD solution.
Also, as I think I noted in an earlier post, one of the qemu
developers has, perhaps independently, implemented the necessary
quirks for AMD cards. His code effects pretty much the same things as
the patch I used. I think there is at least a patch against mainline
qemu in qemu-devel, if it has not been incorporated already. He has
also been doing a fair amount of PCIe work, which I think may be
important for NVidia, assuming their use of extended config space is
causing some of the difficulties.
It would be nice to see enough code get into Xen 4.3 to get AMD cards
working, but with the switch away from qemu-traditional, which is what
the patch is for, this may now be more reliant on the work already
being done in qemu-devel being mainlined into qemu.
- Eric
On Wed, Apr 3, 2013 at 4:45 AM, David TECHER <davidtecher@yahoo.fr>
wrote:> Thanks for clarifying.
>
> I am applying the original patches for nvidia. That''s the reason
why I''ve
> got the 3GM limitation.
>
> Question 1:
>
> In your link
>
> http://lists.xen.org/archives/html/xen-users/2013-02/msg00410.html
>
> Did you apply the following patch
>
> ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch
>
> on Xen 4.3 unstable or Xen 4.2 stable?
>
> Question 2:
>
> Do you use this script
>
> ftp://ftp.enjellic.com/pub/xen/run-passthrough
>
> to start/stop your domU
>
> Question 3
>
> Can you show a domU config file?
>
> Thanks for letting me know.
>
>
>
> ________________________________
> De : Eric Shelton <eshelton@pobox.com>
> À : "xen-users@lists.xen.org" <xen-users@lists.xen.org>;
Patrick Proniewski
> <patpro@patpro.net>; davidtecher@yahoo.fr
> Envoyé le : Mercredi 3 avril 2013 1h26
> Objet : Re:
[Xen-users] PCI/VGA passthrough: differences between Xen
and> ESXi?
>
> David:
> What are the circumstances in which your comment about "Xen - limited
> to 3GB for RAM" applies? Is it in the case in which one uses multiple
> GPU cards?
>
> There are a number of HVM tutorial examples with well over 3GB:
> http://wiki.xen.org/wiki/Secondary_GPU_Passthrough (6GB Win7 HVM with
> passthrough of ATI 6970 and USB controller)
>
http://wiki.xen.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial
> (6GB Win7 HVM with passthrough of ATI 68XX and multiple USB
> controllers)
> None of these tutorials mention a 3GB limitation.
>
> The only recent reference to a
3GB HVM issue during passthrough is here:> http://comments.gmane.org/gmane.comp.emulators.xen.user/77214
> which appears to link the issue with use of nvidia patches from here:
>
http://www.davidgis.fr/download/xen-4.2_rev25240_gfx-passthrough-patchs.tar.bz2
> From what I gather, nvidia cards perform a bit of undocumented voodoo
> (by design) that has not been fully accounted for.
>
> Elsewhere:
> http://new-wiki.xen.org/old-wiki/xenwiki/XenPCIpassthrough.html
> it sounds like there was a PV domain issue arising from e820 for which
> a reliable workaround was incorporated into the mainline code
some> time ago.
>
>
> Patrick:
> From the above examples, it seems like >3GB may be fine if you use
> ATI/AMD GPUs. I have had fairly good experiences with passthrough of
> a single AMD 6570 with the patch mentioned here:
> http://lists.xen.org/archives/html/xen-users/2013-02/msg00410.html
> although I did not explore how much or how little memory I could
> assign to it. Note, however, that patch may make certain assumptions,
> such as that only one GPU is being passed through. I''m not sure
there
> is any unpatched release version (for example, 4.2.1) that reliably
> does GPU passthrough (PCI passthrough of less wonky devices has worked
> for some time).
>
> I think the proposed multi-GPU setup may be putting you out on the
> bleeding
edge (on the other hand, it sounds like it is absolutely
no> available under ESXi). It''s probably not impossible. Maybe
someone
> has already done it. However, how interested and/or comfortable are
> you with code & configuration tweaking, debug, and experimentation?
>
> - Eric
>
>>> Patrick,
>>>
>>> 1) Xen''s vocabulary
>>>
>>> dom0 = the main (first) hypervisor, hosting all virtual machines --
like
>>> your Mac OS X
>>> domU = any virtual machine.
>>>
>>> This is not the real meaning but here I try to stay
"understandable". But
>>> for better understanding please refer to
http://en.wikipedia.org/wiki/Xen
>>>
>>> My understanding is that your dom0 has to be replaced by Linux so
you
can>>> run Xen. But for this part I am not a Mac OS X expert.
>>>
>>> 2) More info - details listed beow are only available for a Xen
dom0
>>> running on Linux.
>>>
>>> [Extracted]:....I''ve ended my testing with ESXi, because I
was not able
>>> to passthrough sound device, and USB....
>>> [Comment]: With Xen you can :)
>>>
>>> [Extracted]: ...with stock ATI Radeon..;
>>> [Comment]...Xen offers VGA PassThrough feature for ATI Radeon.
>>>
>>> Here is a link to a Youtube video to run Crysis 3 on a domU Windows
7 64
>>> Bits with HD 7970
>>>
>>>
http://www.youtube.com/watch?v=GTnchEG4YtI&feature=player_embedded
>>>
>>> As you can
see>>>
>>> - I can use PCI Passthrough to add a XBOX 360 Paddle while playing
Crysis
>>> 3 (see at 09:12 in the Video)
>>>
>>> - I have sound too :)
>>>
>>> But it works on Linux domU too. Here is another Youtube video link
>>>
>>>
http://www.youtube.com/watch?feature=player_embedded&v=KzqOIMaBgX0
>>>
>>> But you need to apply a few patches to able this feature with Xen
4.2 and
>>> over. Not very complicated to do.
>>> Here are my instructions to do it but I know that there are
currently
>>> better way to test it
>>>
>>>
>>>
http://www.davidgis.fr/blog/index.php?2013/03/13/935-xen-43-vga-passthrough-ati-card-hd-7970-changeset-26706
>>>
>>> [Extracted]...memory allocation
>>> [Comment]...Xen - limited to 3GB for RAM. A few developers are
working on
>>> overpassing this limitation.
>>>
>>> [Extracted]...no snaphost possible
>>> [Comment]...If you used Xen with LVM is then snahshotting is easier
to
>>> manage ;)
>>>
>>> [Extracted] no sleep
>>> [Comment] works on Xen for both Linux and Windows. The main issue
is that
>>> you can reboot a domU without restarting the dom0.
>>>
>>>
>>> [Extracted] Is it possible to create a virtualized desktop with VGA
and
>>> PCI passthrough for 2 or 3 VM''s running simultaneously
(ie. each one with
>>> its own video
card)?>>> [Comment] Yes it is doable. Never tested but from experience share
with
>>> other Xen users I know that it should be run fine.
>>>
>>> Hope it helps.
>>>
>>>
>>> Kind regards.
>>>
>>> David.
>
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users
Patrick Proniewski
2013-Apr-04 15:05 UTC
Re: PCI/VGA passthrough: differences between Xen and ESXi?
On 03 avr. 2013, at 01:26, Eric Shelton wrote:> There are a number of HVM tutorial examples with well over 3GB: > http://wiki.xen.org/wiki/Secondary_GPU_Passthrough (6GB Win7 HVM with > passthrough of ATI 6970 and USB controller) > http://wiki.xen.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial > (6GB Win7 HVM with passthrough of ATI 68XX and multiple USB > controllers) > None of these tutorials mention a 3GB limitation.That''s great. Thank you for the pointers. I''ve been into some trouble supporting a >2GB config with GPU passthrough on ESXi, but I''ve been able to work around with 2 lines of configuration. That''s not exactly a surprise for me that GPU passthrough can be quite sensitive to the amount of RAM...> Patrick: > From the above examples, it seems like >3GB may be fine if you use > ATI/AMD GPUs. I have had fairly good experiences with passthrough of > a single AMD 6570 with the patch mentioned here: > http://lists.xen.org/archives/html/xen-users/2013-02/msg00410.html > although I did not explore how much or how little memory I could > assign to it. Note, however, that patch may make certain assumptions, > such as that only one GPU is being passed through. I''m not sure there > is any unpatched release version (for example, 4.2.1) that reliably > does GPU passthrough (PCI passthrough of less wonky devices has worked > for some time). > > I think the proposed multi-GPU setup may be putting you out on the > bleeding edge (on the other hand, it sounds like it is absolutely no > available under ESXi). It''s probably not impossible. Maybe someone > has already done it.I''m not interested in passing through more than 1 graphics card per VM, and the best solution for me would be to pass through 1 graphics card to few VMs (I don''t want to suffer from the noise, the heat, the cost, the power consumption of an array of graphics cards). Technically, ESXi allows you to passthrough only 1 graphics card per VM, but I find no evidence that it won''t allow passing through N graphics cards to N VMs. Of course, low powered fanless graphics cards do exit, and I mould probably be pretty happy with them. Problem is, they use huge coolers, and you can''t put many of them into a single enclosure.> However, how interested and/or comfortable are > you with code & configuration tweaking, debug, and experimentation?I''m highly competent in various IT fields, including system administration on FreeBSD/Unix. But the more I''m getting old, the less I want to spend my nights debugging config and playing with patches :) Ten years ago I''d have setup a test box and played with Xen to achieve my goals. Now I''m more into finding a mature (or at least less experimental) solution. My project is not urgent, I can wait one or two years without any problem. My Mac Pro has still few years of 24/24 operation before needing replacement. When this time will come, I''ll consider a good hardware platform for home virtualisation (Supermicro''s whisper-quiet workstation is my current choice). I''ll keep an eye on Xen, and may be give it a fast try in the next months, but probably not something as complete as my attempts with ESXi. More importantly, Xen would require I switch from the macintosh world to the hackintosh world. Thanks again for your replies, it helps me figuring out the state of GPU passthrough on Xen. Patrick
As of the 3GB limitation, I would like to confirm that for Intel graphics passthrough, there is such limitation for win7 domU. Unfortunately, the reason is not known yet and there seems to be no developer working on this issue. Everything else works just fine -- patches required, though. On Thu, Apr 4, 2013 at 11:05 PM, Patrick Proniewski <patpro@patpro.net> wrote:> On 03 avr. 2013, at 01:26, Eric Shelton wrote: > >> There are a number of HVM tutorial examples with well over 3GB: >> http://wiki.xen.org/wiki/Secondary_GPU_Passthrough (6GB Win7 HVM with >> passthrough of ATI 6970 and USB controller) >> http://wiki.xen.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial >> (6GB Win7 HVM with passthrough of ATI 68XX and multiple USB >> controllers) >> None of these tutorials mention a 3GB limitation. > > That''s great. Thank you for the pointers. > I''ve been into some trouble supporting a >2GB config with GPU passthrough on ESXi, but I''ve been able to work around with 2 lines of configuration. That''s not exactly a surprise for me that GPU passthrough can be quite sensitive to the amount of RAM... > > >> Patrick: >> From the above examples, it seems like >3GB may be fine if you use >> ATI/AMD GPUs. I have had fairly good experiences with passthrough of >> a single AMD 6570 with the patch mentioned here: >> http://lists.xen.org/archives/html/xen-users/2013-02/msg00410.html >> although I did not explore how much or how little memory I could >> assign to it. Note, however, that patch may make certain assumptions, >> such as that only one GPU is being passed through. I''m not sure there >> is any unpatched release version (for example, 4.2.1) that reliably >> does GPU passthrough (PCI passthrough of less wonky devices has worked >> for some time). >> >> I think the proposed multi-GPU setup may be putting you out on the >> bleeding edge (on the other hand, it sounds like it is absolutely no >> available under ESXi). It''s probably not impossible. Maybe someone >> has already done it. > > I''m not interested in passing through more than 1 graphics card per VM, and the best solution for me would be to pass through 1 graphics card to few VMs (I don''t want to suffer from the noise, the heat, the cost, the power consumption of an array of graphics cards). > Technically, ESXi allows you to passthrough only 1 graphics card per VM, but I find no evidence that it won''t allow passing through N graphics cards to N VMs. > Of course, low powered fanless graphics cards do exit, and I mould probably be pretty happy with them. Problem is, they use huge coolers, and you can''t put many of them into a single enclosure. > > >> However, how interested and/or comfortable are >> you with code & configuration tweaking, debug, and experimentation? > > I''m highly competent in various IT fields, including system administration on FreeBSD/Unix. But the more I''m getting old, the less I want to spend my nights debugging config and playing with patches :) > Ten years ago I''d have setup a test box and played with Xen to achieve my goals. Now I''m more into finding a mature (or at least less experimental) solution. > My project is not urgent, I can wait one or two years without any problem. My Mac Pro has still few years of 24/24 operation before needing replacement. When this time will come, I''ll consider a good hardware platform for home virtualisation (Supermicro''s whisper-quiet workstation is my current choice). > I''ll keep an eye on Xen, and may be give it a fast try in the next months, but probably not something as complete as my attempts with ESXi. More importantly, Xen would require I switch from the macintosh world to the hackintosh world. > > Thanks again for your replies, it helps me figuring out the state of GPU passthrough on Xen. > > Patrick > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users
Casey DeLorme
2013-Apr-04 16:43 UTC
Re: PCI/VGA passthrough: differences between Xen and ESXi?
Hi Patrick, I don''t mean to jump in after you already got the information you wanted, but I thought I would throw in my two cents in hopes that it helps save you some time & effort.> > There are a number of HVM tutorial examples with well over 3GB: > > http://wiki.xen.org/wiki/Secondary_GPU_Passthrough (6GB Win7 HVM with > > passthrough of ATI 6970 and USB controller) > > > http://wiki.xen.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial > > (6GB Win7 HVM with passthrough of ATI 68XX and multiple USB > > controllers) > > None of these tutorials mention a 3GB limitation. > > That''s great. Thank you for the pointers. > I''ve been into some trouble supporting a >2GB config with GPU passthrough > on ESXi, but I''ve been able to work around with 2 lines of configuration. > That''s not exactly a surprise for me that GPU passthrough can be quite > sensitive to the amount of RAM... > > I wrote that second guide, it''s a bit out of date so if you choose to useit know that things are a little different. I have been using AMD passthrough on Windows 7 and Windows 8 since September, but I have yet to get it working in Linux, but I blame my own inexperience for that. There are links on my guide to youtube video walkthrough & very short HVM performance demo. Just following the instructions will have you up and running in about 2 hours, but actually understanding how it all works may take quite a bit longer.> > > Patrick: > > From the above examples, it seems like >3GB may be fine if you use > > ATI/AMD GPUs. I have had fairly good experiences with passthrough of > > a single AMD 6570 with the patch mentioned here: > > http://lists.xen.org/archives/html/xen-users/2013-02/msg00410.html > > although I did not explore how much or how little memory I could > > assign to it. Note, however, that patch may make certain assumptions, > > such as that only one GPU is being passed through. I''m not sure there > > is any unpatched release version (for example, 4.2.1) that reliably > > does GPU passthrough (PCI passthrough of less wonky devices has worked > > for some time). > > > > I think the proposed multi-GPU setup may be putting you out on the > > bleeding edge (on the other hand, it sounds like it is absolutely no > > available under ESXi). It''s probably not impossible. Maybe someone > > has already done it. > > I''m not interested in passing through more than 1 graphics card per VM, > and the best solution for me would be to pass through 1 graphics card to > few VMs (I don''t want to suffer from the noise, the heat, the cost, the > power consumption of an array of graphics cards). > Technically, ESXi allows you to passthrough only 1 graphics card per VM, > but I find no evidence that it won''t allow passing through N graphics cards > to N VMs. > Of course, low powered fanless graphics cards do exit, and I mould > probably be pretty happy with them. Problem is, they use huge coolers, and > you can''t put many of them into a single enclosure. > > I have shared the same card by shutting down one instance and bootinganother, but there is a performance degradation problem which I would say is due to the lack of FLR in consumer graphics cards. There are work-arounds but they will feel "hacky", just be ready to deal with major pains getting multiple machines using the same card and make plenty of backups. I don''t know how ESXi handles this problem, but if you do I would love to know more.> > > However, how interested and/or comfortable are > > you with code & configuration tweaking, debug, and experimentation? > > I''m highly competent in various IT fields, including system administration > on FreeBSD/Unix. But the more I''m getting old, the less I want to spend my > nights debugging config and playing with patches :) > Ten years ago I''d have setup a test box and played with Xen to achieve my > goals. Now I''m more into finding a mature (or at least less experimental) > solution. > My project is not urgent, I can wait one or two years without any problem. > My Mac Pro has still few years of 24/24 operation before needing > replacement. When this time will come, I''ll consider a good hardware > platform for home virtualisation (Supermicro''s whisper-quiet workstation is > my current choice). > I''ll keep an eye on Xen, and may be give it a fast try in the next months, > but probably not something as complete as my attempts with ESXi. More > importantly, Xen would require I switch from the macintosh world to the > hackintosh world. > > I use a macbook so I know where you are coming from, I totally prefer themac/unix environment for a workstation. Unfortunately I am pretty sure that OS X requires a UEFI bootloader, and that only exists in upstream-qemu which to my knowledge does not yet work with VGA Passthrough. You would be stuck in SDL or VNC, or maybe ssh, not that those are necessarily bad. Good luck with your adventures, ~Casey _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Patrick Proniewski
2013-Apr-04 17:18 UTC
Re: PCI/VGA passthrough: differences between Xen and ESXi?
Hi Casey, On 04 avr. 2013, at 18:43, Casey DeLorme wrote:> I have shared the same card by shutting down one instance and booting another, but there is a performance degradation problem which I would say is due to the lack of FLR in consumer graphics cards. There are work-arounds but they will feel "hacky", just be ready to deal with major pains getting multiple machines using the same card and make plenty of backups. I don''t know how ESXi handles this problem, but if you do I would love to know more.As far as I remember (my tests are few months old), as soon as you passthrough a PCI device into an ESXi VM, it''s flagged into ESXi as no longer available, so you can''t create another VM using the same device, even if the first one is shutdown. The deal is probably very different with special device like Nvidia''s VGX K2, but I don''t think it''s real passthrough, it''s more likely delegation of GPU power.> I use a macbook so I know where you are coming from, I totally prefer the mac/unix environment for a workstation. Unfortunately I am pretty sure that OS X requires a UEFI bootloader, and that only exists in upstream-qemu which to my knowledge does not yet work with VGA Passthrough. You would be stuck in SDL or VNC, or maybe ssh, not that those are necessarily bad.You can do GPU passthrough in a virtualized Mac OS X on top of ESXi pretty easily. The only limitation ("legal", not technical), is to run this VM on top of a supported Mac (Mac Pro or XServe - haha.). Many users have successfully ran Mac OS X VMs on ESXi on top of Mac Mini or other Macs. I''ve made some benches: <http://www.patpro.net/blog/index.php/2012/12/15/2292-mac-os-x-benchmark-native-vs-virtualized-part-2/> <http://www.patpro.net/blog/index.php/2012/12/17/2342-benchmark-virtualized-os-x-vs-windows/> Now, if I switch to non-Apple hardware, I''ll enter the world of hackintosh. Many users are running Mac OS X on non-Apple hardware, some are running Mac OS X VMs on top of VMware products, and I''ve even found some example of users running an "hackitoshed" Mac OS X on top of Xen with or without VGA passthrough.> Good luck with your adventures,thanks! Patrick
David TECHER
2013-Apr-04 23:19 UTC
Re: PCI/VGA passthrough: differences between Xen and ESXi?
FOR XEN 4.3 UNSTABLE WE CAN NOW START A WINDOWS 7 64 (ATI/VGA PASSTHROUGH)
WITH MORE THAN 3GB OF RAM.
For the current xen-unstable 4.3, applying this patch
ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch
Linux: I was able to start a Ubuntu domU with more than 3GB.
Windows: The only issue was for a Windows 7 domU with more than 3GB
The good new is that if this patch
http://marc.info/?l=qemu-devel&m=136177475215360&q=raw is used too then
we can have more than 3GB of RAM for Windows 7 64 Bits
1) Download Xen sources
======================rev=267773
hg clone -r $rev http://xenbits.xensource.com/staging/xen-unstable.hg/
xen-unstable.hg-rev-XXXXX
cd xen-unstable.hg-rev-XXXXX
2) Configure
===========CURL=$(which curl-config) XML=$(which xml2-config) ./configure
3) Make a first build for tools and cleanup the folder
=====================================================
cd tools
make -j4
# Clean up the folder
make clean
4) Download and apply patches
============================
# Download and apply the 1st patch
cd qemu-xen-dir-remote/
wget "http://marc.info/?l=qemu-devel&m=136177475215360&q=raw"
-O - | patch -p1
# Download and apply the 2nd patch
cd ../..
wget ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch -O - | sed
-e "s:qemu-xen-traditional:qemu-xen-traditional-dir-remote:g" | patch
-p1
5) Build xen and tools
======================cd ..
make -j4 xen && make -j4 tools
6) Build stubdom
===============make stubdom
You shoul have this error
/opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/mini-os-x86_64-ioemu/mini-os.o: In
function `ati_hw_out'':
/opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/ioemu/hw/pt-graphics.c:82: undefined
reference to `iopl''
/opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/ioemu/hw/pt-graphics.c:84: undefined
reference to `iopl''
/opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/mini-os-x86_64-ioemu/mini-os.o: In
function `ati_hw_in'':
/opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/ioemu/hw/pt-graphics.c:72: undefined
reference to `iopl''
/opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/ioemu/hw/pt-graphics.c:74: undefined
reference to `iopl''
I don''t know if commenting iopl(3) and iopl(0) is the good solution but
it works for me
sed -i "s: iopl(: //iopl(:g"
./tools/qemu-xen-traditional-dir-remote/hw/pt-graphics.c
Then rebuild
make stubdom
7) Install all
=============
rm -rf stubdom/gmp-x86_64/
make install-xen && make install-tools PYTHON_PREFIX_ARG= &&
make install-stubdom
8) Linux script to manage domU
=============================Download the Linux shell script
ftp://ftp.enjellic.com/pub/xen/run-passthrough and updated it to your own needs
________________________________
De : David TECHER <davidtecher@yahoo.fr>
À : Eric Shelton <eshelton@pobox.com>
Cc : "xen-users@lists.xen.org" <xen-users@lists.xen.org>;
Patrick Proniewski <patpro@patpro.net>
Envoyé le : Jeudi 4 avril 2013 3h00
Objet : Re: [Xen-users] PCI/VGA passthrough: differences between Xen and ESXi?
I did it myself :)
Indeed this patch seems to be very interesting.
rev=26771
hg clone -r $rev http://xenbits.xensource.com/staging/xen-unstable.hg/
xen-unstable.hg-rev-${rev}
./configure
cd tools
make
make clean
cd ..
wget ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch -O - | sed
-e "s:qemu-xen-traditional:qemu-xen-traditional-dir-remote:g" | patch
-p1
make xen && make install-xen
make tools && make install-tools
Thanks for your help about iopl(). Agreed that iopl(3) and iopl(0) for stubdom
are a bit complicated. I commented out these functions so I can build stubdom
(shame on me!)
make stubdom
make install-stubdom
Using the script to download I was able (after testing/updating only the start
process)
- to start my domU (Linux Ubuntu 12.04 32). I was able to test it for RAM over
than 3GB :). 6GB and 6 vcpus. All my games with Wine works perfectly, VLC,
smplayer are OK :)
- to start my domU (Windows 7). I didn''t test it for RAM over than 3GB.
I will do a full test by reinstalling a Windows 7 64 Bits.
Thanks for helping.
________________________________
De : David TECHER <davidtecher@yahoo.fr>
À : Eric Shelton <eshelton@pobox.com>
Cc : "xen-users@lists.xen.org" <xen-users@lists.xen.org>;
Patrick Proniewski <patpro@patpro.net>
Envoyé le : Mercredi 3 avril 2013 21h49
Objet : Re: [Xen-users] PCI/VGA passthrough: differences between Xen and ESXi?
Thanks for this wonderful story but
I was asking
1) for your revision number/changeset for Xen 4.3 (xl info should return
something)
hg clone -r ????
./configure
make ???
patch -p1 <????
make ???
2) any domU configuration file ?
________________________________
De : Eric Shelton <eshelton@pobox.com>
À : David TECHER <davidtecher@yahoo.fr>
Cc : "xen-users@lists.xen.org" <xen-users@lists.xen.org>;
Patrick Proniewski <patpro@patpro.net>
Envoyé le : Mercredi 3 avril 2013 18h01
Objet : Re: [Xen-users] PCI/VGA passthrough: differences between Xen and ESXi?
1) I applied the patch to 4.3 unstable (back in February). I recall
it applying cleanly (thanks to Dr. Wettstein''s work in updating the
original patches). However, there is a build issue when the patched
files are used to build minios, as its libraries do not provide
iopl(). On my initial pass, I may have #ifdef-ed out the iopl() calls
during minios build time, on the presumption that this code path is
not exercised by minios. However, not being fully confident in that
presumption, in some later work I identified a more appropriate
mechanism: an
existing hypervisor call for iopl. I think revised code
was written, but remains untested.
2) I am using what I saw as the core aspects of that script:
unbind_devices(), re-enabling the devices, and the use of vbetool to
reinitialize the display (which returns the text console in my case).
Although I am doing passthrough of a USB3 controller as well, I did
not bother with rebinding the USB driver, as I simply surrendered use
of those ports for Windows. I also am not waiting for the VM to exit;
instead, I split it the single script into before and after scripts
that were manually invoked.
I think the runtime unbind, bind, and vbetool, versus giving the PCI
devices over to pciback at boot time, makes a difference in terms of
whether you can stop & restart the passthrough VM. With this config,
it is no problem for me. I heard many reports of people starting up
the Windows VM
once, but not again without a reboot.
3+) Tonight I can pull together what I am using and post it up here.
Very little modification of Dr. Wettstein''s patch was done. However,
at this stage, it seems to be the little things that make the
difference between GPU passthrough working or not.
My understanding is that there are many intentional quirks in GPUs to
be in compliance with certain HDCP requirements. Unfortunately, I
think the universe of possible secret register access sequences, etc.
will make it impossible, even with 1:1 mapping in addition, to 100%
convince the display driver, which would be needed to get to
virtualized HDCP compliance (I am interested in a virtualized HTPC
environment). It sounds like, from what I have heard of your work,
that NVidia''s quirks are especially difficult to deal with (not
necessarily intentionally). It was enough to make me abandon
by
existing NVidia GPU in favor of an AMD solution.
Also, as I think I noted in an earlier post, one of the qemu
developers has, perhaps independently, implemented the necessary
quirks for AMD cards. His code effects pretty much the same things as
the patch I used. I think there is at least a patch against mainline
qemu in qemu-devel, if it has not been incorporated already. He has
also been doing a fair amount of PCIe work, which I think may be
important for NVidia, assuming their use of extended config space is
causing some of the difficulties.
It would be nice to see enough code get into Xen 4.3 to get AMD cards
working, but with the switch away from qemu-traditional, which is what
the patch is for, this may now be more reliant on the work already
being done in qemu-devel being mainlined into qemu.
- Eric
On Wed, Apr 3, 2013 at 4:45 AM, David TECHER <davidtecher@yahoo.fr>
wrote:> Thanks for clarifying.
>
> I am applying the original patches for nvidia. That''s the reason
why I''ve
> got the 3GM limitation.
>
> Question 1:
>
> In your link
>
> http://lists.xen.org/archives/html/xen-users/2013-02/msg00410.html
>
> Did you apply the following patch
>
> ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch
>
> on Xen 4.3 unstable or Xen 4.2 stable?
>
> Question 2:
>
> Do you use this script
>
> ftp://ftp.enjellic.com/pub/xen/run-passthrough
>
> to start/stop your domU
>
> Question 3
>
> Can you show a domU config file?
>
> Thanks for letting me know.
>
>
>
> ________________________________
> De : Eric Shelton <eshelton@pobox.com>
> À : "xen-users@lists.xen.org" <xen-users@lists.xen.org>;
Patrick Proniewski
> <patpro@patpro.net>; davidtecher@yahoo.fr
> Envoyé le : Mercredi 3 avril 2013 1h26
> Objet : Re:
[Xen-users] PCI/VGA passthrough: differences between Xen
and> ESXi?
>
> David:
> What are the circumstances in which your comment about "Xen - limited
> to 3GB for RAM" applies? Is it in the case in which one uses multiple
> GPU cards?
>
> There are a number of HVM tutorial examples with well over 3GB:
> http://wiki.xen.org/wiki/Secondary_GPU_Passthrough (6GB Win7 HVM with
> passthrough of ATI 6970 and USB controller)
>
http://wiki.xen.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial
> (6GB Win7 HVM with passthrough of ATI 68XX and multiple USB
> controllers)
> None of these tutorials mention a 3GB limitation.
>
>
The only recent reference to a
3GB HVM issue during passthrough is here:> http://comments.gmane.org/gmane.comp.emulators.xen.user/77214
> which appears to link the issue with use of nvidia patches from here:
>
http://www.davidgis.fr/download/xen-4.2_rev25240_gfx-passthrough-patchs.tar.bz2
> From what I gather, nvidia cards perform a bit of undocumented voodoo
> (by design) that has not been fully accounted for.
>
> Elsewhere:
> http://new-wiki.xen.org/old-wiki/xenwiki/XenPCIpassthrough.html
> it sounds like there was a PV domain issue arising from e820 for which
> a reliable workaround was
incorporated into the mainline code
some> time ago.
>
>
> Patrick:
> From the above examples, it seems like >3GB may be fine if you use
> ATI/AMD GPUs. I have had fairly good experiences with passthrough of
> a single AMD 6570 with the patch mentioned here:
> http://lists.xen.org/archives/html/xen-users/2013-02/msg00410.html
> although I did not explore how much or how little memory I could
> assign to it. Note, however, that patch may make certain assumptions,
> such as that only one GPU is being passed through. I''m not sure
there
> is any unpatched release version (for example, 4.2.1) that reliably
> does GPU passthrough (PCI passthrough of less wonky devices has worked
> for some time).
>
> I think the proposed multi-GPU setup may be putting you out on
the> bleeding
edge (on the other hand, it sounds like it is absolutely
no> available under ESXi). It''s probably not impossible. Maybe
someone
> has already done it. However, how interested and/or comfortable are
> you with code & configuration tweaking, debug, and experimentation?
>
> - Eric
>
>>> Patrick,
>>>
>>> 1) Xen''s vocabulary
>>>
>>> dom0 = the main (first) hypervisor, hosting all virtual machines --
like
>>> your Mac OS X
>>> domU = any virtual machine.
>>>
>>> This is not the real meaning but here I try to stay
"understandable". But
>>> for better understanding please refer to
http://en.wikipedia.org/wiki/Xen
>>>
>>> My understanding is that your dom0 has to be replaced by Linux so
you
can>>> run Xen. But for this part I am not a Mac OS X expert.
>>>
>>> 2) More info - details listed beow are only available for a Xen
dom0
>>> running on Linux.
>>>
>>> [Extracted]:....I''ve ended my testing with ESXi, because I
was not able
>>> to passthrough sound device, and USB....
>>> [Comment]: With Xen you can :)
>>>
>>> [Extracted]: ...with stock ATI Radeon..;
>>> [Comment]...Xen offers VGA PassThrough feature for ATI Radeon.
>>>
>>> Here is a link to a Youtube video to run Crysis 3 on a domU Windows
7 64
>>> Bits with HD 7970
>>>
>>>
http://www.youtube.com/watch?v=GTnchEG4YtI&feature=player_embedded
>>>
>>>
As you can
see>>>
>>> - I can use PCI Passthrough to add a XBOX 360 Paddle while playing
Crysis
>>> 3 (see at 09:12 in the Video)
>>>
>>> - I have sound too :)
>>>
>>> But it works on Linux domU too. Here is another Youtube video link
>>>
>>>
http://www.youtube.com/watch?feature=player_embedded&v=KzqOIMaBgX0
>>>
>>> But you need to apply a few patches to able this feature with Xen
4.2 and
>>> over. Not very complicated to do.
>>> Here are my instructions to do it but I know that there are
currently
>>> better way to test it
>>>
>>>
>>>
http://www.davidgis.fr/blog/index.php?2013/03/13/935-xen-43-vga-passthrough-ati-card-hd-7970-changeset-26706
>>>
>>> [Extracted]...memory allocation
>>> [Comment]...Xen - limited to 3GB for RAM. A few developers are
working on
>>> overpassing this limitation.
>>>
>>> [Extracted]...no snaphost possible
>>> [Comment]...If you used Xen with LVM is then snahshotting is easier
to
>>> manage ;)
>>>
>>> [Extracted] no sleep
>>> [Comment] works on Xen for both Linux and Windows. The main issue
is that
>>> you can reboot a domU without restarting the dom0.
>>>
>>>
>>> [Extracted] Is it possible to create a virtualized desktop with VGA
and
>>> PCI
passthrough for 2 or 3 VM''s running simultaneously (ie. each one
with>>> its own video
card)?>>> [Comment] Yes it is doable. Never tested but from experience share
with
>>> other Xen users I know that it should be run fine.
>>>
>>> Hope it helps.
>>>
>>>
>>> Kind regards.
>>>
>>> David.
>
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users