I have been trying to setup GPU passthrough for a couple of days now with little luck. I''m hoping someone can shed some light as to where I may be going wrong or at least identify some genuine bugs. Essentially pci passthrough works for me but gpu passthrough doesn''t. My system is currently configured as follows (please ask if you need further details) Asrock Z77 e-Itx (stock bios, not sure the version but I can find out. vt-d is enabled) i7 3770S AMD Radeon HD 7750 Dom0 - Ubuntu server 12.04 with desktop environment and the following extra packages installed before the build build-essential zlib1g-dev python-dev libncurses5-dev libssl-dev libx11-dev uuid-dev libyajl-dev libaio-dev libglib2.0-dev pkg-config bridge-utils iproute udev bison flex gettext bin86 bcc iasl markdown ocaml-nox ocaml-findlib git gcc-multilib texinfo pciutils-dev gawk libcurl3 libcurl4-openssl-dev bzip2 module-init-tools transfig tgif texlive-latex-base texlive-latex-recommended texlive-fonts-extra texlive-fonts-recommended mercurial make gcc libc6-dev python python-twisted patch libsdl-dev libjpeg62-dev libbz2-dev e2fslibs-dev git-core xz-utils liblzma-dev liblzo2-dev libvncserver-dev I''ve used the stock Ubuntu provided kernel which is version 3.2.0-27-generic. Dom0 is connected to a display via the intel integrated graphics. xen 4.2 has been build from xen unstable tag 4.2.0-rc1 changeset 25689 using the following commands ./configure make world make install I had to manually add a line to /etc/fstab to get /proc/xen to mount on startup Modules xen-evtchn, xen-gntdev and xen-pciback were setup to load on boot in /etc/modules I''ve then setup xencommons to start on boot: "update-rc.d xencommons defaults 19 18" Grub2 has then been setup to automatically boot the xen kernel. I''ve also had to add the option xsave=0 to the xen boot command line to get things to boot On restart everything looks good. "xl list" returns the Dom0 domain only. "cat /proc/xen/capabilities" returns control_d. I''ve created a windows 7 domU without any pci passthrough and successfully installed windows 7 ultimate. The config file looks like: builder=''hvm'' vcpus = 4 memory = 8192 shadow_memory = 48 name = "xenwin7" vif = [ ''bridge=br0'' ] acpi = 1 apic = 1 disk = [ ''file:/usr/local/xenImages/xenwin7.img,hda,w''] boot="c" sdl=0 vnc=1 vncconsole=1 vncpasswd='''' viridian=1 usb=1 serial=''pty'' usbdevice=''tablet'' on_poweroff="destroy" on_reboot="restart" on_crash="destroy" I would like to be able to pass through the HD 7750 and a USB controller to the VM. I see from lspci there are three devices I need to pass through, two for the gpu (the card itseft and the hd audio for hdmi device) and one for the USB controller. The bus/device ids are: 0000:01:00.0 0000:01:00.1 0000:00:14.0 There are other usb controllers but I would like to leave them with dom0. I can successfully configure the devices for passthrough using "xl pci-assignable-add". In the config I then add the line: pci=["01:00.0","01:00.1","00:14.0"] When I boot the VM everything seems to work fine. Windows picks up all three devices and I can install drivers for them. The USB controller works flawlessly and I can use an attached mouse and keyboard. The radeon card requires a restart after which an attached display comes to life and I have 3D acceleration. Restarting the VM seems to work OK. If however I shut down the VM I have no end of problems. On any subsequent startup the vm struggles to get past the windows splash screen, waiting much longer than usual. During this period dom0 is sluggish regarding mouse and keyboard input even though cpu and memory usage are very low. When windows finally loads I have no active display and I have to view the VM via VNC. In device manage I find that windows has disabled the GPU saying there are not enough resources to run the card. From this point onwards I can do nothing to get the gpu working again aside from removing the device, manually deleting the drivers, and starting again. Note that at this point I am only trying to do secondary passthrough though I would ideally like to get to the point of doing primary passthrough. Adding the line gfx_passthru=1 at any point in all this to the machine config however just prevents the VM from booting entirely; when I VNC in all I get is a qemu prompt and I never get any ouput to the real display. Any suggestions as to how to get this to work would be greatly appreciated as I''ve hit a bit of a brick wall. I should also say that I have managed to get secondary passthrough working using Debian Wheezy and the repository version of xen 4.1. In that case though Dom0 didn''t boot reliably. Sincerely, Matthew Dean _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Wed, Aug 08, 2012 at 02:26:33PM +0100, Matthew Dean wrote:> > I''ve used the stock Ubuntu provided kernel which is version > 3.2.0-27-generic. Dom0 is connected to a display via the intel integrated > graphics. >You might want to try Linux 3.4.x or 3.5.x dom0 kernels aswell. I think there has been some fixes in xen-pciback.> > When I boot the VM everything seems to work fine. Windows picks up all > three devices and I can install drivers for them. The USB controller > works flawlessly and I can use an attached mouse and keyboard. The radeon > card requires a restart after which an attached display comes to life and > I have 3D acceleration. Restarting the VM seems to work OK. If however I > shut down the VM I have no end of problems. On any subsequent startup the > vm struggles to get past the windows splash screen, waiting much longer > than usual. During this period dom0 is sluggish regarding mouse and > keyboard input even though cpu and memory usage are very low. When > windows finally loads I have no active display and I have to view the VM > via VNC. In device manage I find that windows has disabled the GPU saying > there are not enough resources to run the card. From this point onwards I > can do nothing to get the gpu working again aside from removing the > device, manually deleting the drivers, and starting again. >Do you get any errors from Xen (xl dmesg), or from dom0 kernel (dmesg) ? Do you have a serial console?> Note that at this point I am only trying to do secondary passthrough > though I would ideally like to get to the point of doing primary > passthrough. Adding the line gfx_passthru=1 at any point in all this to > the machine config however just prevents the VM from booting entirely; > when I VNC in all I get is a qemu prompt and I never get any ouput to the > real display. >AMD/ATI primary passthru requires extra patches to Xen qemu-dm, those are not included out-of-the-box yet.> Any suggestions as to how to get this to work would be greatly appreciated > as I''ve hit a bit of a brick wall. I should also say that I have managed > to get secondary passthrough working using Debian Wheezy and the > repository version of xen 4.1. In that case though Dom0 didn''t boot > reliably. >-- Pasi
Konrad Rzeszutek Wilk
2012-Aug-09 16:31 UTC
Re: GPU passthrough with Xen 4.2 on Ubuntu 12.04
On Wed, Aug 08, 2012 at 02:26:33PM +0100, Matthew Dean wrote:> I have been trying to setup GPU passthrough for a couple of days now > with little luck. I''m hoping someone can shed some light as to > where I may be going wrong or at least identify some genuine bugs. > Essentially pci passthrough works for me but gpu passthrough > doesn''t. > > My system is currently configured as follows (please ask if you need > further details) > > Asrock Z77 e-Itx (stock bios, not sure the version but I can find > out. vt-d is enabled) > i7 3770S > AMD Radeon HD 7750 > > Dom0 - Ubuntu server 12.04 with desktop environment and the > following extra packages installed before the build > > build-essential zlib1g-dev python-dev libncurses5-dev libssl-dev > libx11-dev uuid-dev libyajl-dev libaio-dev libglib2.0-dev pkg-config > bridge-utils iproute udev bison flex gettext bin86 bcc iasl markdown > ocaml-nox ocaml-findlib git gcc-multilib texinfo pciutils-dev gawk > libcurl3 libcurl4-openssl-dev bzip2 module-init-tools transfig tgif > texlive-latex-base texlive-latex-recommended texlive-fonts-extra > texlive-fonts-recommended mercurial make gcc libc6-dev python > python-twisted patch libsdl-dev libjpeg62-dev libbz2-dev > e2fslibs-dev git-core xz-utils liblzma-dev liblzo2-dev > libvncserver-dev > > I''ve used the stock Ubuntu provided kernel which is version > 3.2.0-27-generic. Dom0 is connected to a display via the intel > integrated graphics. > > xen 4.2 has been build from xen unstable tag 4.2.0-rc1 changeset > 25689 using the following commands > > ./configure > make world > make install > > I had to manually add a line to /etc/fstab to get /proc/xen to mount > on startup > Modules xen-evtchn, xen-gntdev and xen-pciback were setup to load on > boot in /etc/modules > I''ve then setup xencommons to start on boot: "update-rc.d xencommons > defaults 19 18" > > Grub2 has then been setup to automatically boot the xen kernel. I''ve > also had to add the option xsave=0 to the xen boot command line to > get things to boot > > On restart everything looks good. "xl list" returns the Dom0 domain > only. "cat /proc/xen/capabilities" returns control_d. I''ve created > a windows 7 domU without any pci passthrough and successfully > installed windows 7 ultimate. The config file looks like: > > builder=''hvm'' > vcpus = 4 > memory = 8192 > shadow_memory = 48 > name = "xenwin7" > vif = [ ''bridge=br0'' ] > acpi = 1 > apic = 1 > disk = [ ''file:/usr/local/xenImages/xenwin7.img,hda,w''] > boot="c" > sdl=0 > vnc=1 > vncconsole=1 > vncpasswd='''' > viridian=1 > usb=1 > serial=''pty'' > usbdevice=''tablet'' > on_poweroff="destroy" > on_reboot="restart" > on_crash="destroy" > > I would like to be able to pass through the HD 7750 and a USB > controller to the VM. I see from lspci there are three devices I > need to pass through, two for the gpu (the card itseft and the hd > audio for hdmi device) and one for the USB controller. The > bus/device ids are: > > 0000:01:00.0 > 0000:01:00.1 > 0000:00:14.0 > > There are other usb controllers but I would like to leave them with > dom0. I can successfully configure the devices for passthrough > using "xl pci-assignable-add". In the config I then add the line: > > pci=["01:00.0","01:00.1","00:14.0"] > > When I boot the VM everything seems to work fine. Windows picks up > all three devices and I can install drivers for them. The USB > controller works flawlessly and I can use an attached mouse and > keyboard. The radeon card requires a restart after which an > attached display comes to life and I have 3D acceleration. > Restarting the VM seems to work OK. If however I shut down the VM I > have no end of problems. On any subsequent startup the vm struggles > to get past the windows splash screen, waiting much longer than > usual. During this period dom0 is sluggish regarding mouse and > keyboard input even though cpu and memory usage are very low. When > windows finally loads I have no active display and I have to view > the VM via VNC. In device manage I find that windows has disabled > the GPU saying there are not enough resources to run the card. From > this point onwards I can do nothing to get the gpu working again > aside from removing the device, manually deleting the drivers, and > starting again. > > Note that at this point I am only trying to do secondary passthrough > though I would ideally like to get to the point of doing primary > passthrough. Adding the line gfx_passthru=1 at any point in all > this to the machine config however just prevents the VM from booting > entirely; when I VNC in all I get is a qemu prompt and I never getI am not sure if this the same problem I''ve been seeing is but every time I restart a Windows VM with GPU passthrough it crashed. I found that if "Eject" the GPU before shutting down it would work just great the next time I started it. Try that and see if that works. This is btw with an unmodified Xen 4.1 and Fedora''s 3.4.something kernel.> any ouput to the real display. > > Any suggestions as to how to get this to work would be greatly > appreciated as I''ve hit a bit of a brick wall. I should also say > that I have managed to get secondary passthrough working using > Debian Wheezy and the repository version of xen 4.1. In that case > though Dom0 didn''t boot reliably. > > Sincerely, > > Matthew Dean> _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Konrad Rzeszutek Wilk
2012-Aug-09 16:32 UTC
Re: GPU passthrough with Xen 4.2 on Ubuntu 12.04
On Wed, Aug 08, 2012 at 04:42:08PM +0300, Pasi Kärkkäinen wrote:> On Wed, Aug 08, 2012 at 02:26:33PM +0100, Matthew Dean wrote: > > > > I''ve used the stock Ubuntu provided kernel which is version > > 3.2.0-27-generic. Dom0 is connected to a display via the intel integrated > > graphics. > > > > You might want to try Linux 3.4.x or 3.5.x dom0 kernels aswell.The v3.5.x has some issues right now with PCI passthrough. Not sure what - but trying to figure it out.> I think there has been some fixes in xen-pciback. > > > > > > When I boot the VM everything seems to work fine. Windows picks up all > > three devices and I can install drivers for them. The USB controller > > works flawlessly and I can use an attached mouse and keyboard. The radeon > > card requires a restart after which an attached display comes to life and > > I have 3D acceleration. Restarting the VM seems to work OK. If however I > > shut down the VM I have no end of problems. On any subsequent startup the > > vm struggles to get past the windows splash screen, waiting much longer > > than usual. During this period dom0 is sluggish regarding mouse and > > keyboard input even though cpu and memory usage are very low. When > > windows finally loads I have no active display and I have to view the VM > > via VNC. In device manage I find that windows has disabled the GPU saying > > there are not enough resources to run the card. From this point onwards I > > can do nothing to get the gpu working again aside from removing the > > device, manually deleting the drivers, and starting again. > > > > Do you get any errors from Xen (xl dmesg), or from dom0 kernel (dmesg) ? > > Do you have a serial console? > > > > Note that at this point I am only trying to do secondary passthrough > > though I would ideally like to get to the point of doing primary > > passthrough. Adding the line gfx_passthru=1 at any point in all this to > > the machine config however just prevents the VM from booting entirely; > > when I VNC in all I get is a qemu prompt and I never get any ouput to the > > real display. > > > > AMD/ATI primary passthru requires extra patches to Xen qemu-dm, > those are not included out-of-the-box yet. > > > Any suggestions as to how to get this to work would be greatly appreciated > > as I''ve hit a bit of a brick wall. I should also say that I have managed > > to get secondary passthrough working using Debian Wheezy and the > > repository version of xen 4.1. In that case though Dom0 didn''t boot > > reliably. > > > > -- Pasi > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel