I''m trying to pass an ATI Radeon HD 6770 to either a Fedora 16 or Win7 domU, but I''m having no luck and I''m hoping someone can help. My setup: - CPU: i7-2600 - VT-d enabled in BIOS - xen-4.1.2 - dom0 kernel 3.4.3 - ''iommu=force'' and xm dmesg showing I/O virtualization working - pciback built as module - regular PCI passthru (of an audio device) to domU working - unbind radeon driver, assign and bind to pciback (in dom0) I''ve tried assigning gfx_passthru to either 0 or 1. In both cases, the domU appears to start without error. However, using "1" xm reports the domain is running, but consuming all CPU resources and it seems to hang. With "0" on a Fedora VM I can use VNC to see lspci output, which shows the device is visible. However, dmesg shows the radeon driver failing to bind: ... radeon 0000:00:05.0: Expecting atombios for evergreen GPU radeon 0000:00:05.0: Fatal error during GPU init [drm] radeon: finishing device [TTM] Memory type 2 has not been initialized radeon 0000:00:05.0: no bo for sa manager vga_switcheroo: disabled radeon: probe of 0000:00:05.0 failed with error -22 ... With "gfx_passthru=0" on a Win7 VM the emulated graphics shows the swirling logo for a while, then a blue screen as the catalyst driver tires to load before boot completes. From reading list archives this sounds not unlike errors others have encountered, but I never saw a solution. Any help or pointers for where I can look for troubleshooting info would be greatly appreciated. Cheers, Chris
[This is a re-send since my last attempt seemed not to reach the list.] I''m trying to pass an ATI graphics adapter to either a Fedora 16 or Win7 domU, but I''m having no luck and I''m hoping someone can help. My setup: - CPU: i7-2600 - VT-d enabled in BIOS - ATI Radeon HD 6770 - xen-4.1.2 - dom0 kernel 3.4.3 - ''iommu=force'' and xm dmesg showing I/O virtualization working - pciback built as module - regular PCI passthru (of an audio device) to domU working - unbind radeon driver, assign and bind to pciback (in dom0) I''ve tried assigning gfx_passthru to either 0 or 1. In both cases, the domU appears to start without error. However, when using "gfx_passthry=1" xm reports the domain is running, but consuming all CPU resources and the instance seems to hang. With "gfx_passthry=0" on a Fedora VM it boots and I can use the emulated graphics (or ssh) to get lspci output, which shows the ATI device is visible. However, dmesg shows the radeon driver failing to bind: ... radeon 0000:00:05.0: Expecting atombios for evergreen GPU radeon 0000:00:05.0: Fatal error during GPU init [drm] radeon: finishing device [TTM] Memory type 2 has not been initialized radeon 0000:00:05.0: no bo for sa manager vga_switcheroo: disabled radeon: probe of 0000:00:05.0 failed with error -22 ... With "gfx_passthru=0" on a Win7 VM the emulated graphics shows the swirling logo for a while, then a blue screen as the catalyst driver tries to load before boot completes. From reading xen-devel and xen-users archives this situation sounds not unlike what others have encountered, but I never saw a solution. Any help or pointers for where I can look for troubleshooting info would be greatly appreciated. Cheers, Chris
On Sat, Jun 23, 2012 at 5:57 PM, Chris <kavefish@gmail.com> wrote:> [This is a re-send since my last attempt seemed not to reach the list.] > > I''m trying to pass an ATI graphics adapter to either a Fedora 16 or Win7 > domU, but I''m having no luck and I''m hoping someone can help. > > My setup: > - CPU: i7-2600 > - VT-d enabled in BIOS > - ATI Radeon HD 6770 > - xen-4.1.2 > - dom0 kernel 3.4.3 > - ''iommu=force'' and xm dmesg showing I/O virtualization working > - pciback built as module > - regular PCI passthru (of an audio device) to domU working > - unbind radeon driver, assign and bind to pciback (in dom0) > > I''ve tried assigning gfx_passthru to either 0 or 1. In both cases, the > domU appears to start without error. However, when using > "gfx_passthry=1" xm reports the domain is running, but consuming all CPU > resources and the instance seems to hang. > > With "gfx_passthry=0" on a Fedora VM it boots and I can use the emulated > graphics (or ssh) to get lspci output, which shows the ATI device is > visible. However, dmesg shows the radeon driver failing to bind: > > ... > radeon 0000:00:05.0: Expecting atombios for evergreen GPU > radeon 0000:00:05.0: Fatal error during GPU init > [drm] radeon: finishing device > [TTM] Memory type 2 has not been initialized > radeon 0000:00:05.0: no bo for sa manager > vga_switcheroo: disabled > radeon: probe of 0000:00:05.0 failed with error -22 > ... > > With "gfx_passthru=0" on a Win7 VM the emulated graphics shows the > swirling logo for a while, then a blue screen as the catalyst driver > tries to load before boot completes. > > From reading xen-devel and xen-users archives this situation sounds not > unlike what others have encountered, but I never saw a solution. Any > help or pointers for where I can look for troubleshooting info would be > greatly appreciated. >In the Linux domU, try install AMD''s own driver (from http://support.amd.com/us/gpudownload/Pages/index.aspx). I have more luck with that one than with the kernel''s default driver.
Thanks for the suggestion. It''s taking me down a weird path. I attempted installation of the fglrx Linux driver, which failed (while building the module for my domU kernel). At first I thought perhaps it failed because an instance of X was running as the Nvidia driver does, so I switched the Fedora VM to run level 3... and then my monitor lit up... the one that the Fedora VM was supposed to control. Never mind there''s supposed to be no X11 in run level 3. As far as I can tell there''s no driver bound to the device. The /sys/bus/pci/device/<BFD>/ directory contains no driver symlink. Also, the lsmod listing doesn''t include anything that''s obviously a video driver. So I''m guessing that''s what happens when it''s just pure software rendering, but I don''t really know. In any case, it''s definitely not either of the radeon or fglrx drivers. The configuration that produces the effect closest to what I was aiming for has ''gfx_passthru=0'', which allows me to watch boot via VNC and then the secondary adapter takes over when gdm starts. However, I use ''gfx_passthru=1'' then I get the same behaviour as before where the guest starts without error, but appears to hang shortly after power-on. The Win7 VM behaviour is unchanged: ''gfx_passthru=0'' starts booting until the ATI driver loads and bluescreens; ''gfx_passthru=1'' hangs very early after power-on. The result is the same whether using the latest official release driver or the latest beta driver. Any other ideas greatly appreciated
On Sat, Jun 23, 2012 at 9:30 PM, Chris <kavefish@gmail.com> wrote:> Thanks for the suggestion. It''s taking me down a weird path. > > I attempted installation of the fglrx Linux driver, which failed > (while building the module for my domU kernel). At first I thought > perhaps it failed because an instance of X was running as the Nvidia > driver does, so I switched the Fedora VM to run level 3... and then my > monitor lit up... the one that the Fedora VM was supposed to control. > Never mind there''s supposed to be no X11 in run level 3. >I''ve had fglrx fail to build the kernel module when using a 3.4 kernel (in the domU), resulting in no 2D or 3D acceleration. It compiles properly with an older kernel, Ubuntu Precise''s current default kernel in my case.> As far as I can tell there''s no driver bound to the device. The > /sys/bus/pci/device/<BFD>/ directory contains no driver symlink. Also, > the lsmod listing doesn''t include anything that''s obviously a video > driver. So I''m guessing that''s what happens when it''s just pure > software rendering, but I don''t really know. In any case, it''s > definitely not either of the radeon or fglrx drivers. > > The configuration that produces the effect closest to what I was > aiming for has ''gfx_passthru=0'', which allows me to watch boot via VNC > and then the secondary adapter takes over when gdm starts. However, I > use ''gfx_passthru=1'' then I get the same behaviour as before where the > guest starts without error, but appears to hang shortly after > power-on. > > The Win7 VM behaviour is unchanged: ''gfx_passthru=0'' starts booting > until the ATI driver loads and bluescreens; ''gfx_passthru=1'' hangs > very early after power-on. The result is the same whether using the > latest official release driver or the latest beta driver. >I don''t have windows working yet either (I''m trying the win8 consumer preview).
Hello, My experience with Windows is that you have to restart the physical system before passing the card to install drivers. If you restart the system you may have to eject it from the system tray icon for safe device ejection. When you pass the card to a DomU, the card is initialized. However, Windows does not reset the card. So after that DomU is shutdown the card remains initialized. When a Windows 7 DomU boots if the card is already in an initialized state, and driver installation, removal, and 3D applications will fail. I attached a diagram as a visual aid to depict what is happening. ~Casey On Sat, Jun 23, 2012 at 3:30 PM, Chris <kavefish@gmail.com> wrote:> Thanks for the suggestion. It''s taking me down a weird path. > > I attempted installation of the fglrx Linux driver, which failed > (while building the module for my domU kernel). At first I thought > perhaps it failed because an instance of X was running as the Nvidia > driver does, so I switched the Fedora VM to run level 3... and then my > monitor lit up... the one that the Fedora VM was supposed to control. > Never mind there''s supposed to be no X11 in run level 3. > > As far as I can tell there''s no driver bound to the device. The > /sys/bus/pci/device/<BFD>/ directory contains no driver symlink. Also, > the lsmod listing doesn''t include anything that''s obviously a video > driver. So I''m guessing that''s what happens when it''s just pure > software rendering, but I don''t really know. In any case, it''s > definitely not either of the radeon or fglrx drivers. > > The configuration that produces the effect closest to what I was > aiming for has ''gfx_passthru=0'', which allows me to watch boot via VNC > and then the secondary adapter takes over when gdm starts. However, I > use ''gfx_passthru=1'' then I get the same behaviour as before where the > guest starts without error, but appears to hang shortly after > power-on. > > The Win7 VM behaviour is unchanged: ''gfx_passthru=0'' starts booting > until the ATI driver loads and bluescreens; ''gfx_passthru=1'' hangs > very early after power-on. The result is the same whether using the > latest official release driver or the latest beta driver. > > Any other ideas greatly appreciated > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On 6/23/12 8:28 PM, Casey DeLorme wrote:> When you pass the card to a DomU, the card is initialized. However, > Windows does not reset the card. So after that DomU is shutdown the > card remains initialized. > > When a Windows 7 DomU boots if the card is already in an initialized > state, and driver installation, removal, and 3D applications will fail.That''s interesting. If I understand, the passed gfx device must be reset before being passed to any domU. What you described leads me to conclude that one of the configuration options (that I''m using) described on the Xen.org wiki may be broken. In my case the dom0 pciback driver is built as a module, so I can''t hide the gfx device from dom0 at boot with xen-pciback.hide. I''m using the late binding approach, which means dom0 binds a driver to the device (e.g. radeon), which I assume has the consequence of initializing the card. Then I manually unbind the dom0 driver and assign the device to pciback. Whether the card is reset during unbind is unclear, but I''d guess not. So, a physical machine reboot currently wouldn''t prevent the gfx device from being initialized before being passed to a domU. (Is anyone using the late binding approach with a gfx device successfully?) The alternative I should try is to use modprobe.conf to blacklist the radeon driver, so only pciback ever binds to the card. I''ll give that a try next. (Is anyone using the modprobe.conf approach with gfx pass-through successfully?) Though one thing about your theory is bugging me. My Linux domU was repeatedly able to drive the gfx device without any physical machine reboots. It had no 2D or 3D acceleration and - perhaps more importantly - no driver was obviously bound to the device. In any case, I''ll give the modprobe.conf approach a try. If that fails I''ll rebuild the kernel to include pciback. (I''d be curious if there''s anyone out there with gfx pass-through working with pciback as a module.) Cheers, Chris
Chris, I do not know if the late-binding method performs an ejection, but your logic is sound. To my knowledge the ejection process is more specific to Windows and not Linux systems, but late ejection is different than a system reboot. In either case, you should be able to use the safely remove hardware option in Windows to manually reset the card after Windows has loaded. However this has to be done before you install or remove any drivers or open 3D applications. If the driver already installed and is crashing, you may have to reinstall Windows. I could post another two pages worth of case-study results in my attempts to fix BSoD''s from failed installs, but trust me when I say a reinstallation is faster. The diagram I made was only to depict Windows instances, as I have not even gotten passthrough working on Linux (currently at "mountall: disconnected from plymouth"). However, from what others have posted Linux performs an FLR and properly resets the card. To my limited understanding FLR or Function Level Reset, is the virtual machine implementation for reseting physical devices, which makes sense since a virtual machine does not change the power-state of devices like a physical reboot would. With Windows I see two possibilities. One is that Windows is issuing an FLR at boot time, but because the device is passed as a secondary graphics card it is not attached when the event is issued. It is equally possible that Windows does not issue an FLR, Microsoft doesn''t exactly advocate running their OS as a virtual machine do they? My speculations could be wildly inaccurate, but given the test results it is logically sound. I encountered a 100% failure rate with driver installation, removal, and 3D applications if the card had already been used once prior to the current system boot. Success during first-use of the card, and also when using Tobias Geiger''s suggestion of the safely eject device feature. If you get your Linux DomU working I wouldn''t mind your notes on the process. ~Casey On Sun, Jun 24, 2012 at 8:43 AM, Chris <kavefish@gmail.com> wrote:> On 6/23/12 8:28 PM, Casey DeLorme wrote: > > When you pass the card to a DomU, the card is initialized. However, > > Windows does not reset the card. So after that DomU is shutdown the > > card remains initialized. > > > > When a Windows 7 DomU boots if the card is already in an initialized > > state, and driver installation, removal, and 3D applications will fail. > > That''s interesting. If I understand, the passed gfx device must be reset > before being passed to any domU. > > What you described leads me to conclude that one of the configuration > options (that I''m using) described on the Xen.org wiki may be broken. > > In my case the dom0 pciback driver is built as a module, so I can''t hide > the gfx device from dom0 at boot with xen-pciback.hide. I''m using the > late binding approach, which means dom0 binds a driver to the device > (e.g. radeon), which I assume has the consequence of initializing the > card. Then I manually unbind the dom0 driver and assign the device to > pciback. Whether the card is reset during unbind is unclear, but I''d > guess not. So, a physical machine reboot currently wouldn''t prevent the > gfx device from being initialized before being passed to a domU. (Is > anyone using the late binding approach with a gfx device successfully?) > > The alternative I should try is to use modprobe.conf to blacklist the > radeon driver, so only pciback ever binds to the card. I''ll give that a > try next. (Is anyone using the modprobe.conf approach with gfx > pass-through successfully?) > > Though one thing about your theory is bugging me. My Linux domU was > repeatedly able to drive the gfx device without any physical machine > reboots. It had no 2D or 3D acceleration and - perhaps more importantly > - no driver was obviously bound to the device. > > In any case, I''ll give the modprobe.conf approach a try. If that fails > I''ll rebuild the kernel to include pciback. (I''d be curious if there''s > anyone out there with gfx pass-through working with pciback as a module.) > > Cheers, > Chris >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users