Hi all, I''m trying, without success, to get VGA Passthrough working. My relevant hardware configuration is: - Intel i7-3770 (supports VT-d) - ASRock Z77 Pro4-M (VT-x and VT-d enabled from BIOS) - AMD Radeon HD7750, enumerated as 01:00.0 (VGA) and 01:00.1 (HD audio) - AMD Radeon HD6450, enumerated as 02:00.0 and 02:00.1 (I physically removed this one on my latest tests) On dom0 I run an Ubuntu Server 12.04.2 LTS 64 bit, kernel version 3.5.0-34.55~precise1, xm tools. I have a Windows 7 32 bit virtual machine that works regularly with gfx_passthru=0 I blacklisted modules radeon and snd_hda_intel. At boot, console is on HD7750, then it switches to onboard Intel HD Graphics, leaving the HD7750 screen black and on. In /etc/rc.local I call a script to bind pciback driver to 01:00.0, 01:00.1, 02:00.0, and 02:00.1 My tests were mainly with HD7750, but I gave some tries also to HD6450 I used: - Ubuntu supplied Xen 4.1.2-2ubuntu2.10 - Xen 4.2.2 built from sources - Xen 4.3-unstable from git HEAD The results: - Xen 4.1.2-2ubuntu2.10 - with gfx_passthru=1 and pci=[''01:00.0''] xm create apparently works but the screen remains off (to be precise, it goes DPMS off on the first run after a reboot and stays off for the followings) - with gfx_passthru=0 and pci=[''01:00.0'',''01:00.1''] xm create fails with "Error: (22, ''Invalid argument'')" (but the screen goes off on the first try after a reboot) - Xen 4.2.2 Build: - git clone git://xenbits.xen.org/xen.git xen-devel - git branch pac-4.2.2 RELEASE-4.2.2 - git checkout pac-4.2.2 - ./configure --prefix=/usr/local - make -j9 world Install: - sudo make -j9 install - replacement of /etc/init.d/xen* with modified ones in order to make everything start on Ubuntu - ldconfig - reboot Results: - with gfx_passthru=0 and pci=[''01:00.0'',''01:00.1''] xm create fails with "Error: (22, ''Invalid argument'')" (but the screen goes off on the first try after a reboot) - with gfx_passthru=0 and pci=[''01:00.0''] xm create works - with gfx_passthru=1 and pci=[''01:00.0''] xm create starts the virtual machine but the screen remains off, VM CPU stays high (~70%) until xm shutdown; if I start again the VM with gfx_passthru=0, it starts and Windows does not complain about being shut down uncleanly (I think that with gfx_passthru=1 guest OS does not even start bootstrapping) - Xen 4.3-unstable Build: - git checkout master - git clean -f -d -x - git reset --hard - git branch pac-4.3-unstable - git checkout pac-4.3-unstable - ./configure - make -j9 world Install: - sudo make -j9 install - replacement of /etc/init.d/xen* with modified ones in order to make everything start on Ubuntu - ldconfig - reboot Results: - with gfx_passthru=0 and without pci=... xm create works, VM starts and works, screen stays black & on - with gfx_passthru=0 and pci=[''01:00.0'',''01:00.1''] xm create works, VM starts and works, screen goes off - with gfx_passthru=1 and pci=[''01:00.0'',''01:00.1''] xm create starts VM, screen goes off, VM CPU goes 70% and stays there; VM does not boot; last message I see with xm dmesg is ROMBIOS Revision "$Revision: 1.221 $ $Date: 2008/12/07 17:32:29 $" I investigated 4.3-unstable. Adding logs to tools/firmware/rombios/rombios.c, I discovered that VM gets stuck in call to VGA ROM init. I also gave a look to the ''Invalid argument'' failure in 4.2.2 Call to pirq_permit_access() on IRQ 17, the interrupt of HD audio at 01.00.1 , returns -EINVAL The failing call chain is: do_domctl(cmd=XEN_DOMCTL_irq_permission, allow=1, pirq=17) -> pirq_permit_access() -> domain_pirq_to_irq() -> pirq_field() -> pirq_info() -> radix_tree_lookup() radix_tree_lookup() fails to find the structure for IRQ 17, while it succeeds finding IRQ 16, belonging to VGA at 01.00.0 I tried also secondary passthrough, but the native driver installation program crashes and shortly afterwards Windows guest goes blue screen. Here is my VM configuration: builder=''hvm'' memory = 1024 vcpus=1 name = "win7" vif = [''bridge=xenbr0''] disk = [''phy:/dev/xen2/win7,hda,w''] acpi = 1 device_model_version = "qemu-xen-traditional" device_model = ''qemu-dm'' stdvga=1 usbdevice=''tablet'' boot="c" sdl=0 serial=''pty'' vnc=1 vnclisten="0.0.0.0" vncpasswd="" vncdisplay="7" xen_platform_pci=1 viridian=1 apic=1 gfx_passthru=1 pci=[''01:00.0'',''01:00.1''] I can provide all the configuration files and logs you desire. I can also try patches, other Xen versions and, if needed, another kernel (even building it from sources), but I fear I''m missing some very basic detail :-/ What are your advices? Cheers, Paolo _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Hi, I run into the same problem with Ubuntu getting Error: (22, ''Invalid argument'') with Ubuntu repository versions. I ended up compiling xen 4.1.4 with patches for VGA Passthrough and I''m actually running 3 windows 7 domUs, 2 of them with VGA passthrough + USB controller and the other one running SQL server. Check this message for info. http://xen.1045712.n5.nabble.com/VGA-passthrough-radeon-4850-as-primary-card-tp5714485p5714606.html I applied those patches and didn''t use GPLPV drivers as I was getting BSOD and Dom0 freezing issues. -- View this message in context: http://xen.1045712.n5.nabble.com/Problems-with-VGA-Passthrough-tp5717358p5717359.html Sent from the Xen - User mailing list archive at Nabble.com.
Hello Javier, On Wed, Jul 3, 2013 at 10:56 PM, dkeruza <dkeruza@gmail.com> wrote:> > I ended up compiling xen 4.1.4 with patches for VGA Passthrough > [...] > Check this message for info. > > http://xen.1045712.n5.nabble.com/VGA-passthrough-radeon-4850-as-primary-card-tp5714485p5714606.html > >following your advice, I was able to see VGA Passthrough working for the very first time. I compiled Xen 4.1.4 after applying the patches here: ftp://ftp.enjellic.com/pub/xen/xen-4.1.3.ati-passthrough.patch The installation was a bit messy, since makefiles of 4.1.4 did not honor PREFIX variable. USB passthrough does not seem to work, I used PCI passthrough of one of the two EHCI controllers on the motherboard. Unfortunately, however, with this setup my VMs are quite unstable. A Windows 8 VM reboots after waiting for some time after boot, before logging in. A Windows 7 VM at first seemed to work, but it stopped responding after being left alone for a few minutes (perhaps it''s something related to power management). Do you know why these patches are not on Xen development tree (yet)? Cheers, Paolo _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On 07/08/2013 05:20 PM, Paolo Cancedda wrote:> Hello Javier, > > On Wed, Jul 3, 2013 at 10:56 PM, dkeruza <dkeruza@gmail.com > <mailto:dkeruza@gmail.com>> wrote: > > > I ended up compiling xen 4.1.4 with patches for VGA Passthrough > [...] > Check this message for info. > http://xen.1045712.n5.nabble.com/VGA-passthrough-radeon-4850-as-primary-card-tp5714485p5714606.html > > > following your advice, I was able to see VGA Passthrough working for the > very first time. > I compiled Xen 4.1.4 after applying the patches here: > ftp://ftp.enjellic.com/pub/xen/xen-4.1.3.ati-passthrough.patch > The installation was a bit messy, since makefiles of 4.1.4 did not honor > PREFIX variable. > USB passthrough does not seem to work, I used PCI passthrough of one of > the two EHCI controllers on the motherboard. > Unfortunately, however, with this setup my VMs are quite unstable. > A Windows 8 VM reboots after waiting for some time after boot, before > logging in. > A Windows 7 VM at first seemed to work, but it stopped responding after > being left alone for a few minutes (perhaps it''s something related to > power management). > Do you know why these patches are not on Xen development tree (yet)? > Cheers,I have it all working stably with 4.2.2 with only the XSA patches (using RPMs from the CRC repository). I am also using PCI passthrough for a USB controller (I have two USB ports on the same physical controller, which is handy as I only have to pass in one USB controller). If you are using > 2GB of RAM in domU this is going to cause you the sort of instability that will drive you nuts - it certainly did for me, and the only reason I persevered in the face of extreme frustration is because KVM didn''t work for me with VGA passthrough _at all_, and ESX was a non-starter due to my hardware''s lack of ACS capable PCI bridges which according to documentation, ESX needs for PCI passthrough. DomUs with > 2GB of RAM are not going to work with passing through devices with large BARs such as GPUs. A fix for this bug is in 4.3, due out any day now. I had mixed luck with ATI passthrough, I''m using quasi-Quadro cards (GeForce modified into Quadro, see here for some details: http://www.altechnative.net/2013/06/23/nvidia-cards-geforce-quadro-and-geforce-modified-into-a-quadro-for-virtualized-gaming/ Since the article I''ve moved further with the quasi-Quadros, I currently have a GTX470 (Quadro 5000) and a GTX480 (Quadro 6000) passed through to two separate VMs, and this is working lovely. No performance degradations or host crashes with domU reboots any more. As far as I can tell, judging by what ended up in dom0 syslog, the crashes were caused by PCI memory overwrites (AER errors all over the logs, search the archives for my previous reports on the subject). I am not sure why the host crashes have actually stopped when rebooting VMs, but when I got the 2nd VGA passthrough domU set up, the problem just went away. All in all, I''m finding quasi-Quadros just work a lot better than ATIs in terms of how they handle domU restarts and re-initialization. I''m currently using an 8800GT card for dom0, but am hoping to switch to an ATI 4850, purely so that I can make my life easier by blacklisting all the nvidia drivers and not having the complication of selective pre-emptive partial detaching of Nvidia devices before the nvidia driver loads. Unfortunately, cards later than ATI 4xxx are unusable for me due to only having one DL-DVI capable port (one of the insurmountable reasons why Nvidia is the only option for me at the moment). Gordan
Did you install the GPLPV drivers? if you did, that is the problem. Also, Make sure you blacklist the radeon driver. When you get a chance, post the lspci list here and the vm config file. I wanna see what the problem could be with your USB controller. Also, make sure the BIOS in your motherboard is updated. -- View this message in context: http://xen.1045712.n5.nabble.com/Problems-with-VGA-Passthrough-tp5717358p5717503.html Sent from the Xen - User mailing list archive at Nabble.com.
On 07/11/2013 06:52 PM, dkeruza wrote:> Did you install the GPLPV drivers? if you did, that is the problem. > Also, Make sure you blacklist the radeon driver. > When you get a chance, post the lspci list here and the vm config file. I > wanna see what the problem could be with your USB controller. > Also, make sure the BIOS in your motherboard is updated.Not sure about the OPs USB controller PCI passthrough issue, but reviewing the configuration changes between when I reliably had a host crash from rebooting domUs with VGA passthrough, the only thing that had changed was the passthrough of USB controllers. Originally I was passing through 2 USB controllers, and immediately after a domU shutdown, I''d get an error in syslog about a missed interrupt on an interrupt shared between a USB controller and something else (I forget what else it was, I can find out later.). It turned out that I had 2 ports on each of the USB controller devices, rather than 1, so I re-arranged the keyboards/mice so that I was passing only one controller with 2 devices to the domU. Ever since I did that the IRQ error went away on domU shutdown/reboot, and the domU rebooting has been working perfectly. The controller I was passing through before is now for dom0''s keyboard and mouse, so I''m pretty sure that it was passthrough of that specific PCI device that was causing the problem (why it would crash the GPU before anything else remains a mystery). If I had to guess, it''s got something to do with IRQ sharing. Note: All USB controllers in question are the ones in the ICH10 SB, but the ones I''m successfully using now and the ones that seemed to be responsible for dom0 crashes on domU reboots. I''m hoping this might be useful to someone as a data point that suggests that the device that seems to crash isn''t necessarily the device that is the root cause of the crash when passing through multiple PCI devices. Gordan