Aloha! Concisely, I''ve been trying for several days to get VGA/PCI passthrough working. Regarding VGA vs PCI passthrough, I''m not entirely certain of the difference? Does Xen perform a different set of operations when passing through a VGA card versus any PCIe card? I''ve read the wiki and it seems the answer is "yes;" however, when passing through a graphics adapter, how would I instruct Xen/qemu-dm that said adapter is a graphics card and not a normal PCIe card? In essence, how does one pass-through a graphics card that is NOT the primary adapter (my BIOS does not boot with it)? My problem is somewhat complex. I have two Radeon cards (one 5870 as primary, and one 5570 as secondary). The BIOS boots with the 5870 and I cannot swap the cards due to space constraints within the case. Linux then boots using the 5870. I''ve hidden the 5570 using pciback (compiled in the kernel not as a module). I can successfully get a Windows 7 32-bit domU to see the card, and install the drivers. However on boot, the system BSODs due to an Error 116 in atikmpag.sys! I can avoid this if I use an older driver (10.x series), without Catalyst Control Center. However, even using the older driver, or the Microsoft driver, neither results in actual display from the graphics card. I get nothing. The most I''ve gotten is a black screen, and then the system will crash. I''ve searched the mailing lists, and I''ve done every permutation of configurations I can think of. I''m stumped! I''m left with the belief that I cannot pass-through the secondary graphics adapter. I need to, because I don''t have a second computer at the moment, and I need the primary adapter to operate dom0. I''ve thought of running Linux off the secondary adapter, but it doesn''t seem possible with the open-source radeon driver and fglrix (closed-source) won''t work with Xen apparently. Am I missing something, here? Any help would be appreciated! -- View this message in context: http://xen.1045712.n5.nabble.com/VGA-PCI-Passthrough-of-Secondary-Graphics-Adapter-tp5715723.html Sent from the Xen - User mailing list archive at Nabble.com.
Hello Alex, To answer your first question, simply provide the BDF (Bus Device Function) from lspci. My understanding is that graphics devices are missing a translation for legacy processes, such as boot code, and this varies by device. I am using the onboard graphics with Dom0 as the primary, and passing a Radeon HD 6870 to a virtual machine. Graphics cards are not "shared", so even if it was your primary card you would have to remove it from Dom0 to pass it. The BSoD you are encountering, does it happen when you reboot? In my experience this is caused by the lack of FLR (Function Level Reset) in consumer cards, the solution is manual ejection and if you install drivers without being aware of the problem it will still experience odd problems. Experience varies however, my advice would be to check the marksmail <http://xen.markmail.org/>search engine for Function Level Reset or Passthrough BSOD. ~Casey On Sat, Apr 27, 2013 at 7:16 PM, Alex Karaoui <code3studios@gmail.com>wrote:> Aloha! > > Concisely, I''ve been trying for several days to get VGA/PCI passthrough > working. Regarding VGA vs PCI passthrough, I''m not entirely certain of the > difference? Does Xen perform a different set of operations when passing > through a VGA card versus any PCIe card? I''ve read the wiki and it seems > the answer is "yes;" however, when passing through a graphics adapter, how > would I instruct Xen/qemu-dm that said adapter is a graphics card and not a > normal PCIe card? > > In essence, how does one pass-through a graphics card that is NOT the > primary adapter (my BIOS does not boot with it)? > > My problem is somewhat complex. I have two Radeon cards (one 5870 as > primary, and one 5570 as secondary). The BIOS boots with the 5870 and I > cannot swap the cards due to space constraints within the case. Linux then > boots using the 5870. I''ve hidden the 5570 using pciback (compiled in the > kernel not as a module). I can successfully get a Windows 7 32-bit domU to > see the card, and install the drivers. > > However on boot, the system BSODs due to an Error 116 in atikmpag.sys! I > can avoid this if I use an older driver (10.x series), without Catalyst > Control Center. However, even using the older driver, or the Microsoft > driver, neither results in actual display from the graphics card. I get > nothing. The most I''ve gotten is a black screen, and then the system will > crash. > > I''ve searched the mailing lists, and I''ve done every permutation of > configurations I can think of. I''m stumped! I''m left with the belief that > I cannot pass-through the secondary graphics adapter. I need to, because I > don''t have a second computer at the moment, and I need the primary adapter > to operate dom0. I''ve thought of running Linux off the secondary adapter, > but it doesn''t seem possible with the open-source radeon driver and fglrix > (closed-source) won''t work with Xen apparently. > > Am I missing something, here? Any help would be appreciated! > > > > -- > View this message in context: > http://xen.1045712.n5.nabble.com/VGA-PCI-Passthrough-of-Secondary-Graphics-Adapter-tp5715723.html > Sent from the Xen - User mailing list archive at Nabble.com. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
1) If passing declaring the BDF is all that is needed to pass a VGA card, in the same respect as an PCI device, then there is no difference between PCI passthrough and VGA passthrough IF the VGA card being passed through is not in use by the dom0? 2) The card I wish to pass is the 5570, in the secondary PCIe slot. It is not being used by the dom0, and is subsequently hidden (xen-pciback.hide). 3) Neither Radeon in my system supports FLR; however, many people use the Radeon with VGA passthrough somewhat routinely. I seem to be having immense trouble getting this to work? I have not gotten anything to display other than a black screen (which does show that something is happening). I just can''t understand why? And regarding the BSODs, when I install the card, I''ve gotten it to say "Functioning normally" in Device Manager, but the screen is disabled in "Screen Resolution," and the when I attempt to "Extend to Display" in order to enable the Radeon monitor, the screen goes blank and the system crashes. Upon reboot, the system BSODs (atikmpag.sys) I''ve tried physically powering down dom0 and bringing it back up, but the error persists. As an aside, I''ve compiled Xen with a patch that supposedly fixes these issues (patches qemu-dm), and I''ve attempted to use the script provided (run-passthrough), but that hasn''t resulted in any change whatsoever. I greatly appreciate your time in assisting me! On Sat, Apr 27, 2013 at 3:30 PM, Casey DeLorme <cdelorme@gmail.com> wrote:> Hello Alex, > > To answer your first question, simply provide the BDF (Bus Device > Function) from lspci. > > My understanding is that graphics devices are missing a translation for > legacy processes, such as boot code, and this varies by device. > > I am using the onboard graphics with Dom0 as the primary, and passing a > Radeon HD 6870 to a virtual machine. Graphics cards are not "shared", so > even if it was your primary card you would have to remove it from Dom0 to > pass it. > > The BSoD you are encountering, does it happen when you reboot? In my > experience this is caused by the lack of FLR (Function Level Reset) in > consumer cards, the solution is manual ejection and if you install drivers > without being aware of the problem it will still experience odd problems. > > Experience varies however, my advice would be to check the marksmail > <http://xen.markmail.org/>search engine for Function Level Reset or > Passthrough BSOD. > > ~Casey > > > On Sat, Apr 27, 2013 at 7:16 PM, Alex Karaoui <code3studios@gmail.com>wrote: > >> Aloha! >> >> Concisely, I''ve been trying for several days to get VGA/PCI passthrough >> working. Regarding VGA vs PCI passthrough, I''m not entirely certain of >> the >> difference? Does Xen perform a different set of operations when passing >> through a VGA card versus any PCIe card? I''ve read the wiki and it seems >> the answer is "yes;" however, when passing through a graphics adapter, how >> would I instruct Xen/qemu-dm that said adapter is a graphics card and not >> a >> normal PCIe card? >> >> In essence, how does one pass-through a graphics card that is NOT the >> primary adapter (my BIOS does not boot with it)? >> >> My problem is somewhat complex. I have two Radeon cards (one 5870 as >> primary, and one 5570 as secondary). The BIOS boots with the 5870 and I >> cannot swap the cards due to space constraints within the case. Linux >> then >> boots using the 5870. I''ve hidden the 5570 using pciback (compiled in the >> kernel not as a module). I can successfully get a Windows 7 32-bit domU >> to >> see the card, and install the drivers. >> >> However on boot, the system BSODs due to an Error 116 in atikmpag.sys! I >> can avoid this if I use an older driver (10.x series), without Catalyst >> Control Center. However, even using the older driver, or the Microsoft >> driver, neither results in actual display from the graphics card. I get >> nothing. The most I''ve gotten is a black screen, and then the system will >> crash. >> >> I''ve searched the mailing lists, and I''ve done every permutation of >> configurations I can think of. I''m stumped! I''m left with the belief >> that >> I cannot pass-through the secondary graphics adapter. I need to, because >> I >> don''t have a second computer at the moment, and I need the primary adapter >> to operate dom0. I''ve thought of running Linux off the secondary adapter, >> but it doesn''t seem possible with the open-source radeon driver and fglrix >> (closed-source) won''t work with Xen apparently. >> >> Am I missing something, here? Any help would be appreciated! >> >> >> >> -- >> View this message in context: >> http://xen.1045712.n5.nabble.com/VGA-PCI-Passthrough-of-Secondary-Graphics-Adapter-tp5715723.html >> Sent from the Xen - User mailing list archive at Nabble.com. >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xen.org >> http://lists.xen.org/xen-users >> > >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
> 1) If passing declaring the BDF is all that is needed to pass a VGA card, > in the same respect as an PCI device, then there is no difference between > PCI passthrough and VGA passthrough IF the VGA card being passed through is > not in use by the dom0? > > From the perspective of Dom0, but graphics (aka VGA) cards from what Ihave read need legacy code support for things like boot-time handling, making it more complex than say network cards, SATA cards, video capture cards, or other types.> 2) The card I wish to pass is the 5570, in the secondary PCIe slot. It is > not being used by the dom0, and is subsequently hidden (xen-pciback.hide). > > AFAIK slot number doesn''t mean anything to your DomU, so once the deviceis passed there is no "primary slot" in the mix.> 3) Neither Radeon in my system supports FLR; however, many people use the > Radeon with VGA passthrough somewhat routinely. I seem to be having > immense trouble getting this to work? I have not gotten anything to > display other than a black screen (which does show that something is > happening). I just can''t understand why? > > I have used a Radeon HD 6870 successfully since August last year in bothWindows 7 and Windows 8 virtual machines. For me the lack of FLR creates problems during: - Driver installation - Driver Upgrades - System Reboots The lack of FLR, to my understanding, prevents the virtual machine from resetting the card state, leaving it initialized when switching or rebooting your DomU. As a result the card is activated multiple times, creating various issues with a range of problems. During my tests, I broke down the steps logically to resolve the issues I had installing and updating drivers. In general, anytime I update or install AMD drivers I would reboot the entire physical machine to ensure the graphics card state was proper. From my experiences this required a freshly installed Windows system, if you are working on a system you already attempted driver installation on and it failed (BSoD''s or otherwise) uninstalling the failed drivers and trying again loops back to the same failures for mostly unknown reasons. Presumable it has to do with the use of the .NET framework for installation, and leaving bits of data behind on failure, but I never confirmed the exact details. For me, once the drivers are properly installed, rebooting the system still leaves the card in a pre-initialized state, which to avoid performance degradation I would manually reset the card using eject media from the lower right icon. This is not a perfect solution, and I do not rely on it when performing driver installation or updates. Note that Windows 8 especially, but also Windows 7, will attempt to automatically install drivers which for me led to reboots that broke my attempted installation. I had to tell the system not to automatically update to work around this during the first installation process. Again, this is my experience, other users have reported vastly different problems with a range of other possible causes and many unsolved.> And regarding the BSODs, when I install the card, I''ve gotten it to say > "Functioning normally" in Device Manager, but the screen is disabled in > "Screen Resolution," and the when I attempt to "Extend to Display" in order > to enable the Radeon monitor, the screen goes blank and the system > crashes. Upon reboot, the system BSODs (atikmpag.sys) I''ve tried > physically powering down dom0 and bringing it back up, but the error > persists. > > As an aside, I''ve compiled Xen with a patch that supposedly fixes these > issues (patches qemu-dm), and I''ve attempted to use the script provided > (run-passthrough), but that hasn''t resulted in any change whatsoever. > >Since we haven''t verified this yet, are you using Xen 4.2 and the xl toolstack? Also, how are you checking the Device Manager? If you are still using SDL or VNC then you are using the emulated graphics, and you cannot extend emulated to physical, you either use one or the other. Try changing the monitor source to use the AMD input, and reboot the system after the drivers are installed and it should automatically begin sending video to your monitor. To get mouse and keyboard to the system pass a USB controller and use new inputs. Technically you can still send input through VNC, but that may not work well. The reboot error is consistent with my diagnosis above, you can try my suggestions but I can by no means claim they will work for you. If you applied the AMD primary patch to Xen source before building, have you added the passthrough flag to your DomU configuration and seeing if that changes anything. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On 04/28/2013 02:30 AM, Casey DeLorme wrote:> The BSoD you are encountering, does it happen when you reboot? In my > experience this is caused by the lack of FLR (Function Level Reset) in > consumer cards,I am not entirely convinced of that, mainly because I have yet to see lspci output showing that Quadro cards have FLreset+. From what I can tell the only difference is the device PCI ID. When it is detected the driver decides that since you paid a 4x premium for the card, it will reset the card for you by wiping the registers when asked to do so. Can somebody demonstrate otherwise, with lspci output showing FLreset+ on a Quadro FX [345]000 or Quadro [246]000 card? As far as I can tell, Nvidia have a reasonably decent driver that handles card reinitialization gracefully in a VM encironment, while ATI have one of far more questionable quality (but more open, it seems). Also, does anyone have an ATI FirePro card that they could post lspci output from? I can believe that those might have FLreset+ to compensate for the driver not being up to the task.> the solution is manual ejection and if you install > drivers without being aware of the problem it will still experience odd > problems.What is this "manual ejection" I see mentioned? How exactly do you do that from dom0?> Experience varies however, my advice would be to check the marksmail > <http://xen.markmail.org/>search engine for Function Level Reset or > Passthrough BSOD.Or just get an Nvidia card modifiable to a Quadro and modify it (or just buy a Quadro if you can afford it and lack the required soldering experience), instead of wasting infeasible amounts time getting an ATI card to work reliably. I have seen a number of posts in the archives from people who have had nothing but problems with ATI cards only to find that a Quadro card works perfectly every time. Gordan
On Sat, 27 Apr 2013 16:16:25 -0700 (PDT), Alex Karaoui <code3studios@gmail.com> wrote:> I''ve thought of running Linux off the secondary adapter, > but it doesn''t seem possible with the open-source radeon driver and > fglrix > (closed-source) won''t work with Xen apparently.You can specify the PCI ID to the driver (both radeon and fglrx drivers, IIRC) to tell it which card to fire up xorg on (you''ll need to create your own xorg.conf. Make sure you blacklist the radeon kernel driver. Do you catually need the fglrx driver in Dom0? Radeon driver does offer basic acceleration. It''s not great, but it''s workable. FWIW, the Nvidia driver _can_ be persuaded to work with Xen in Dom0 (see: http://www.altechnative.net/2013/04/14/wquxga-a-k-a-omgwtf-ibm-t221-3840x2400-204dpi-monitor-part-6-regressing-drivers-and-xen/ ) if you are considering a switch. Gordan
On 04/28/2013 07:11 AM, Casey DeLorme wrote:> The lack of FLR, to my understanding, prevents the virtual machine from > resetting the card state, leaving it initialized when switching or > rebooting your DomU. As a result the card is activated multiple times, > creating various issues with a range of problems.You see, I''m not actually convinced FLR is such a vitally important feature. All you are really doing with FLR is pushing the re-initialization responsibility onto the BIOS of the PCI device. Without it, the driver has to do it. What is it that the driver cannot do to reset the device? Think low-level - all you are really doing is putting values into the registers of the GPU.> During my tests, I broke down the steps logically to resolve the issues > I had installing and updating drivers. In general, anytime I update or > install AMD drivers I would reboot the entire physical machine to ensure > the graphics card state was proper.Interestingly, the one thing that various people have been saying about ATI cards is that they get slower and slower on every guest reboot, unless you reboot the host. I haven''t seen any evidence of this on my 6450. The fps I get in the likes of OCCT and Furmark are the same after the guest has been rebooted a few times. The problem I have is that an application entering full-screen mode causes a crash.> From my experiences this required a freshly installed Windows system, > if you are working on a system you already attempted driver installation > on and it failed (BSoD''s or otherwise) uninstalling the failed drivers > and trying again loops back to the same failures for mostly unknown > reasons. Presumable it has to do with the use of the .NET framework for > installation, and leaving bits of data behind on failure, but I never > confirmed the exact details.I don''t know if that helped in any way, but I uninstalled anything remotely .NET-ish, including the client profile.> For me, once the drivers are properly installed, rebooting the system > still leaves the card in a pre-initialized state, which to avoid > performance degradation I would manually reset the card using eject > media from the lower right icon. This is not a perfect solution, and I > do not rely on it when performing driver installation or updates.Where is this "eject" for a PCI device? What distro are you actually running? Gordan
Hi Gordan, Sorry for not getting back to you sooner, it''s a lot of information to sift through.> The lack of FLR, to my understanding, prevents the virtual machine from >> resetting the card state, leaving it initialized when switching or >> rebooting your DomU. As a result the card is activated multiple times, >> creating various issues with a range of problems. >> > > You see, I''m not actually convinced FLR is such a vitally important > feature. All you are really doing with FLR is pushing the re-initialization > responsibility onto the BIOS of the PCI device. Without it, the driver has > to do it. What is it that the driver cannot do to reset the device? Think > low-level - all you are really doing is putting values into the registers > of the GPU. > >The problems I encountered during my tests would point to the fact that the card state is not being reset, and FLR happens to exist to do just that for virtualization. It seems like a fairly logical assessment to say that because FLR is not available for consume cards that this is a problem. It is very possible that it could be something else, but I am just pointing at the most likely cause using what I know personally.> During my tests, I broke down the steps logically to resolve the issues >> I had installing and updating drivers. In general, anytime I update or >> install AMD drivers I would reboot the entire physical machine to ensure >> the graphics card state was proper. >> > > Interestingly, the one thing that various people have been saying about > ATI cards is that they get slower and slower on every guest reboot, unless > you reboot the host. I haven''t seen any evidence of this on my 6450. The > fps I get in the likes of OCCT and Furmark are the same after the guest has > been rebooted a few times. The problem I have is that an application > entering full-screen mode causes a crash. > >When I reboot my system the card still works just fine for Windows, but will perform very poorly when I run a 3D application. To resolve this I eject the device which causes it to reset and restores the performance, for the duration it runs. Rebooting again will require the same steps. However, I have not experienced a gradual decrease in performance from continual reboots, possibly because I eject and reset it after rebooting. The tests I mentioned were a large number of fresh Windows installations and AMD driver installations during the 11.x-12.x versions. Out of the first 16 installations 13 failed to install AMD drivers when faced with a rebooted Windows and a card with a state that had not been reset. The 3 that succeeded BSoD''d on reboot or randomly shortly after login. Another 30 installations with a fresh system, only rebooted during the installation process claimed to work successfully but roughly 15% of the time would experience video tearing and BSoD without warning (on first boot), and almost always on reboot. The final 22 installations I rebooted the whole machine during both stages of installation, and ran Windows 7 with that for four months without a problem, the migrated to Windows 8 which has been running for three months.> From my experiences this required a freshly installed Windows system, >> if you are working on a system you already attempted driver installation >> on and it failed (BSoD''s or otherwise) uninstalling the failed drivers >> and trying again loops back to the same failures for mostly unknown >> reasons. Presumable it has to do with the use of the .NET framework for >> installation, and leaving bits of data behind on failure, but I never >> confirmed the exact details. >> > > I don''t know if that helped in any way, but I uninstalled anything > remotely .NET-ish, including the client profile. > >I spent three days trying to figure out the specifics, and did the same as you, and that did not solve the problems. In using .NET it either left something else on the system or I missed something, but I was more concerned with getting the system working than figuring out why it was breaking, and three days was enough time to reinstall several times.> For me, once the drivers are properly installed, rebooting the system >> still leaves the card in a pre-initialized state, which to avoid >> performance degradation I would manually reset the card using eject >> media from the lower right icon. This is not a perfect solution, and I >> do not rely on it when performing driver installation or updates. >> > > Where is this "eject" for a PCI device? What distro are you actually > running? > >The ejection process has nothing to do with Linux or Dom0. I call it a "manual ejection" as it does not occur automatically. The ejection takes place from inside the running Windows virtual machine, after it has restarted. There is an icon to eject media in the bottom menu, when clicked it displays any passed devices. My understanding is this triggers a hot-plug ejection on the device, if you know how to reproduce that in Linux let me know. I point my fingers at FLR because its described responsibility is to reset device state in virtual environments, and every one of the problems I have encountered appear to be linked to the state of the card. However, other users have posted very different experiences, leading me to believe that it could well be hardware specific. I started with a consumer nVidia card, it didn''t work that''s why I switched to AMD. If you can share the model of a $180 USD consumer nVidia card with HDMI out and onboard audio that can be converted to a working Quadro model card and is equal to or outperforms my AMD I would gladly switch. From what I have seen the Quadro 2000 is the earliest model with onboard audio, and none of them have HDMI out so you would need a DisplayPort to HDMI adapter, plus they cost double what I paid for my card. If you compare passmark benchmarks my card has a 60% performance gain over a Quadro K2000, and 30% over the Quadro 4000 which costs four times as much. Going back to the lack of demonstration videos really makes me wary of throwing that much money into a "possible" alternative, especially one that performs worse than my current. While I have seen emails mentioning Quadro, that''s about it. No instructions or demonstration videos or performance comparisons. Makes me a bit wary about dropping four times the cost of my current card for less performance when the only supporting documents are various emails and a wiki page. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On 04/29/2013 08:31 PM, Casey DeLorme wrote:> The lack of FLR, to my understanding, prevents the virtual > machine from > resetting the card state, leaving it initialized when switching or > rebooting your DomU. As a result the card is activated multiple > times, > creating various issues with a range of problems. > > > You see, I'm not actually convinced FLR is such a vitally important > feature. All you are really doing with FLR is pushing the > re-initialization responsibility onto the BIOS of the PCI device. > Without it, the driver has to do it. What is it that the driver > cannot do to reset the device? Think low-level - all you are really > doing is putting values into the registers of the GPU. > > > The problems I encountered during my tests would point to the fact that > the card state is not being reset, and FLR happens to exist to do just > that for virtualization. It seems like a fairly logical assessment to > say that because FLR is not available for consume cards that this is a > problem. It is very possible that it could be something else, but I am > just pointing at the most likely cause using what I know personally.I'm not disputing that FLR makes it easier. What I am saying, however, is that if the driver is written without cutting corners and making assumptions, it should be able to manually re-initialize the device.> During my tests, I broke down the steps logically to resolve the > issues > I had installing and updating drivers. In general, anytime I > update or > install AMD drivers I would reboot the entire physical machine > to ensure > the graphics card state was proper. > > > Interestingly, the one thing that various people have been saying > about ATI cards is that they get slower and slower on every guest > reboot, unless you reboot the host. I haven't seen any evidence of > this on my 6450. The fps I get in the likes of OCCT and Furmark are > the same after the guest has been rebooted a few times. The problem > I have is that an application entering full-screen mode causes a crash. > > > When I reboot my system the card still works just fine for Windows, but > will perform very poorly when I run a 3D application. To resolve this I > eject the device which causes it to reset and restores the performance, > for the duration it runs. Rebooting again will require the same steps. > However, I have not experienced a gradual decrease in performance from > continual reboots, possibly because I eject and reset it after rebooting. > The tests I mentioned were a large number of fresh Windows installations > and AMD driver installations during the 11.x-12.x versions. > > Out of the first 16 installations 13 failed to install AMD drivers when > faced with a rebooted Windows and a card with a state that had not been > reset. The 3 that succeeded BSoD'd on reboot or randomly shortly after > login. > > Another 30 installations with a fresh system, only rebooted during the > installation process claimed to work successfully but roughly 15% of the > time would experience video tearing and BSoD without warning (on first > boot), and almost always on reboot. > > The final 22 installations I rebooted the whole machine during both > stages of installation, and ran Windows 7 with that for four months > without a problem, the migrated to Windows 8 which has been running for > three months.With the reliability record you describe above, that makes VGA passthrough stability as close to useless as it can possibly get. I'm going to try to snipe a suitable Quadro on eBay for a sane sum in the next week or two (I'm not sure what I mean by that, when we are actually talking about what is essentially a GTX260 for the same money as a GTX660, but you have to start somewhere, I suppose). It will be interesting to see if that "just works". If it does, and the Quadro also shows up as lacking FLreset, then that puts the onus purely on driver quality.> From my experiences this required a freshly installed Windows > system, > if you are working on a system you already attempted driver > installation > on and it failed (BSoD's or otherwise) uninstalling the failed > drivers > and trying again loops back to the same failures for mostly unknown > reasons. Presumable it has to do with the use of the .NET > framework for > installation, and leaving bits of data behind on failure, but I > never > confirmed the exact details. > > > I don't know if that helped in any way, but I uninstalled anything > remotely .NET-ish, including the client profile. > > > I spent three days trying to figure out the specifics, and did the same > as you, and that did not solve the problems. In using .NET it either > left something else on the system or I missed something, but I was more > concerned with getting the system working than figuring out why it was > breaking, and three days was enough time to reinstall several times.I went the other way - installed all the updates, including .NET ones, then removed them. But after upgrading the Xen stack to 4.2.2 from 4.2.1 (keeping 4.2.1-6 hypervisor, later ones just error out when starting the VM with "unknown parameter"), it all seemed to magically start working (except for full-screen 3D not working at all).> For me, once the drivers are properly installed, rebooting the > system > still leaves the card in a pre-initialized state, which to avoid > performance degradation I would manually reset the card using eject > media from the lower right icon. This is not a perfect > solution, and I > do not rely on it when performing driver installation or updates. > > > Where is this "eject" for a PCI device? What distro are you actually > running? > > > The ejection process has nothing to do with Linux or Dom0. I call it a > "manual ejection" as it does not occur automatically. The ejection > takes place from inside the running Windows virtual machine, after it > has restarted. There is an icon to eject media in the bottom menu, when > clicked it displays any passed devices. My understanding is this > triggers a hot-plug ejection on the device, if you know how to reproduce > that in Linux let me know.Uhh... You are saying that the ATI card shows up as an ejectable device in the Windows VM? Really?? /me goes to check Typical - after the VM booted perfectly every time for the last 2 days, now I get nothing but a BSOD (failed attempt to reset the display driver) every time - and I'm doing this on a freshly rebooted machine.> I point my fingers at FLR because its described responsibility is to > reset device state in virtual environments, and every one of the > problems I have encountered appear to be linked to the state of the > card. However, other users have posted very different experiences, > leading me to believe that it could well be hardware specific.I suspect it's down to shoddy drivers. Has anyone actually reported problems with a supported Quadro card ([246]000 or FX [345]800)?> I started with a consumer nVidia card, it didn't work that's why I > switched to AMD. If you can share the model of a $180 USD consumer > nVidia card with HDMI out and onboard audio that can be converted to a > working Quadro model card and is equal to or outperforms my AMD I would > gladly switch.GTX680 and GTX690 have been done: http://www.eevblog.com/forum/projects/hacking-nvidia-cards-into-their-professional-counterparts/msg207550/#msg207550 Note: Small amount of soldering required. But they don't fit in the $180 budget envelope. It does look very much like most Nvidia cards are modifiable to equivalent quadros (those that have reasonably equivalent Quadros at least). I'm just getting a genuine low-end Quadro (for hopefully <= £130) first to make sure it'll work, before I get something higher end for modifying. I'm vaguely tempted by a Titan, but the problem is that the only equivalent enterprise grade card is a Tesla, which means no video out, which would somewhat defeat the purpose. Then again, there seems to be evidence that the number of shaders doesn't have to match with what the Quadro model is expected to have - it looks like the hardware capabilities get auto-deteced correctly.> From what I have seen the Quadro 2000 is the earliest > model with onboard audio, and none of them have HDMI out so you would > need a DisplayPort to HDMI adapter, plus they cost double what I paid > for my card. If you compare passmark benchmarks my card has a 60% > performance gain over a Quadro K2000, and 30% over the Quadro 4000 which > costs four times as much. Going back to the lack of demonstration > videos really makes me wary of throwing that much money into a > "possible" alternative, especially one that performs worse than my > current. While I have seen emails mentioning Quadro, that's about it.Expect a report back from me on this as soon as I get my hands on a Quadro (2000 or FX3800, I have no intention of spending more than a bare minimum). My priority is to have something that works, and works _reliably_.> No instructions or demonstration videos or performance comparisons. > Makes me a bit wary about dropping four times the cost of my current > card for less performance when the only supporting documents are various > emails and a wiki page.Let me flip that one around - there are plenty of blog entries and videos of ATI cards working, and yet there are several recent posts on this list about using ATI cards and VGA passthrough working at best unreliably, and more often not working at all. Don't fall into the "It must be true, I read it on the internet!" trap. :) Gordan _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
> I''m not disputing that FLR makes it easier. What I am saying, however, is > that if the driver is written without cutting corners and making > assumptions, it should be able to manually re-initialize the device. > > I totally wish that was the case, if you find a linux method of doing thisI would love to know about it.> With the reliability record you describe above, that makes VGA passthrough > stability as close to useless as it can possibly get. > >The installation process is super buggy, but after getting it installed correctly I have had it working just fine for months. Keep in mind I was running Wheezy (unstable) and Xen 4.2 (unstable a the time), so I was more concerned with "can it work" than "is it stable".> Uhh... You are saying that the ATI card shows up as an ejectable device in > the Windows VM? Really?? > > /me goes to check > > Typical - after the VM booted perfectly every time for the last 2 days, > now I get nothing but a BSOD (failed attempt to reset the display driver) > every time - and I''m doing this on a freshly rebooted machine. > >I posted a youtube video <http://www.youtube.com/watch?v=s1Ro0KxHooQ>showing the whole process, I demonstrate ejection @21:25> I point my fingers at FLR because its described responsibility is to >> reset device state in virtual environments, and every one of the >> problems I have encountered appear to be linked to the state of the >> card. However, other users have posted very different experiences, >> leading me to believe that it could well be hardware specific. >> > > I suspect it''s down to shoddy drivers. Has anyone actually reported > problems with a supported Quadro card ([246]000 or FX [345]800)? > > Not the mailing lists, but as supported devices (as per the wiki) I''mwilling to bet nVidia has been contacted for troubleshooting support.> I started with a consumer nVidia card, it didn''t work that''s why I >> switched to AMD. If you can share the model of a $180 USD consumer >> nVidia card with HDMI out and onboard audio that can be converted to a >> working Quadro model card and is equal to or outperforms my AMD I would >> gladly switch. >> > > GTX680 and GTX690 have been done: > > http://www.eevblog.com/forum/**projects/hacking-nvidia-cards-** > into-their-professional-**counterparts/msg207550/#**msg207550<http://www.eevblog.com/forum/projects/hacking-nvidia-cards-into-their-professional-counterparts/msg207550/#msg207550> > > Note: Small amount of soldering required. > > I saw that, would be awesome if I had a bit of extra cash, but more ifsomeone can confirm that it works.> But they don''t fit in the $180 budget envelope. It does look very much > like most Nvidia cards are modifiable to equivalent quadros (those that > have reasonably equivalent Quadros at least). > > I''m just getting a genuine low-end Quadro (for hopefully <= £130) first to > make sure it''ll work, before I get something higher end for modifying. > > I''m vaguely tempted by a Titan, but the problem is that the only > equivalent enterprise grade card is a Tesla, which means no video out, > which would somewhat defeat the purpose. Then again, there seems to be > evidence that the number of shaders doesn''t have to match with what the > Quadro model is expected to have - it looks like the hardware capabilities > get auto-deteced correctly. > > > From what I have seen the Quadro 2000 is the earliest >> model with onboard audio, and none of them have HDMI out so you would >> need a DisplayPort to HDMI adapter, plus they cost double what I paid >> for my card. If you compare passmark benchmarks my card has a 60% >> performance gain over a Quadro K2000, and 30% over the Quadro 4000 which >> costs four times as much. Going back to the lack of demonstration >> videos really makes me wary of throwing that much money into a >> "possible" alternative, especially one that performs worse than my >> current. While I have seen emails mentioning Quadro, that''s about it. >> > > Expect a report back from me on this as soon as I get my hands on a Quadro > (2000 or FX3800, I have no intention of spending more than a bare minimum). > My priority is to have something that works, and works _reliably_. > > That is something to look forward to, please do send out details if youget a Quadro and it works. If it does no longer having to eject on reboots might be worth the drop in performance.> > No instructions or demonstration videos or performance comparisons. >> Makes me a bit wary about dropping four times the cost of my current >> card for less performance when the only supporting documents are various >> emails and a wiki page. >> > > Let me flip that one around - there are plenty of blog entries and videos > of ATI cards working, and yet there are several recent posts on this list > about using ATI cards and VGA passthrough working at best unreliably, and > more often not working at all. Don''t fall into the "It must be true, I read > it on the internet!" trap. :) > > A good rule to follow to be sure, and that''s why I like detailedsupporting information and demonstrations. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On 04/29/2013 10:17 PM, Casey DeLorme wrote:> > I''m not disputing that FLR makes it easier. What I am saying, > however, is that if the driver is written without cutting corners > and making assumptions, it should be able to manually re-initialize > the device. > > I totally wish that was the case, if you find a linux method of doing > this I would love to know about it.The reality is that this doesn''t really help us - if the driver is poorly written and can''t handle re-initializing the device on it''s own, it doesn''t really matter whether the device has FLR or not. IMO the problem is the quality of the driver, not the implementation of FLR. Years of experience have taught me that as much as Nvidia''s drivers suck, ATI''s drivers suck even more. :(> I point my fingers at FLR because its described responsibility is to > reset device state in virtual environments, and every one of the > problems I have encountered appear to be linked to the state of the > card. However, other users have posted very different experiences, > leading me to believe that it could well be hardware specific. > > > I suspect it''s down to shoddy drivers. Has anyone actually reported > problems with a supported Quadro card ([246]000 or FX [345]800)? > > Not the mailing lists, but as supported devices (as per the wiki) I''m > willing to bet nVidia has been contacted for troubleshooting support.If only ATI provided a method to contact them for such support, considering they claim that they "tested" 4850/4870 cards to work with Xen and VGA passthrough. If they''ve tested it, it''d be nice if they documented it. Personally I''ve not seen any evidence of the sort of stability you''d rightfully expect when the manufacturer vouches that something works.> I started with a consumer nVidia card, it didn''t work that''s why I > switched to AMD. If you can share the model of a $180 USD consumer > nVidia card with HDMI out and onboard audio that can be > converted to a > working Quadro model card and is equal to or outperforms my AMD > I would > gladly switch. > > > GTX680 and GTX690 have been done: > > http://www.eevblog.com/forum/__projects/hacking-nvidia-cards-__into-their-professional-__counterparts/msg207550/#__msg207550 > <http://www.eevblog.com/forum/projects/hacking-nvidia-cards-into-their-professional-counterparts/msg207550/#msg207550> > > Note: Small amount of soldering required. > > I saw that, would be awesome if I had a bit of extra cash, but more if > someone can confirm that it works.As I said, give me a week or two to get my hands on a Quadro, then another week or two to get my hands on a GTX680 and modify it. This WILL happen because it will take some pretty insurmountable obstacles to make me concede to live with rebooting my machine whenever I want to have a blast of Borderlands 2 with friends. :)> From what I have seen the Quadro 2000 is the earliest > model with onboard audio, and none of them have HDMI out so you > would > need a DisplayPort to HDMI adapter, plus they cost double what I > paid > for my card. If you compare passmark benchmarks my card has a 60% > performance gain over a Quadro K2000, and 30% over the Quadro > 4000 which > costs four times as much. Going back to the lack of demonstration > videos really makes me wary of throwing that much money into a > "possible" alternative, especially one that performs worse than my > current. While I have seen emails mentioning Quadro, that''s > about it. > > > Expect a report back from me on this as soon as I get my hands on a > Quadro (2000 or FX3800, I have no intention of spending more than a > bare minimum). My priority is to have something that works, and > works _reliably_. > > That is something to look forward to, please do send out details if you > get a Quadro and it works.Count on it. Gordan
Just out of interest, do you have the gplpv drivers installed on your Windows DomU? I currently have a desktop machine with two VGA cards (Nvidia GF550Ti available to Dom0 and an AMD Radeon 6770 passed through to a Windows 7 DomU) and Windows would keeo BSOD''ing on bootup if I also had the gplpv drivers installed. After I managed to uninstall them, I never reinstalled them as the basic emulated performance was good enough for me. David _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On 04/30/2013 06:38 AM, Peregrine wrote:> Just out of interest, do you have the gplpv drivers installed on your > Windows DomU? I currently have a desktop machine with two VGA cards > (Nvidia GF550Ti available to Dom0 and an AMD Radeon 6770 passed through > to a Windows 7 DomU) and Windows would keeo BSOD''ing on bootup if I also > had the gplpv drivers installed. After I managed to uninstall them, I > never reinstalled them as the basic emulated performance was good enough > for me.I never installed the PV drivers - I read about possible issues it can cause and figured I can look into it later, if I ever get everything else working stably. Gordan
I have been using version 0.11.0.357 for three months on Windows 8 with passthrough. It makes a significant difference in disk (SSD) performance, but I am not sure about network. As Gordan said, not exactly the best choice for stability yet. On Tue, Apr 30, 2013 at 3:11 AM, Gordan Bobic <gordan@bobich.net> wrote:> On 04/30/2013 06:38 AM, Peregrine wrote: > >> Just out of interest, do you have the gplpv drivers installed on your >> Windows DomU? I currently have a desktop machine with two VGA cards >> (Nvidia GF550Ti available to Dom0 and an AMD Radeon 6770 passed through >> to a Windows 7 DomU) and Windows would keeo BSOD''ing on bootup if I also >> had the gplpv drivers installed. After I managed to uninstall them, I >> never reinstalled them as the basic emulated performance was good enough >> for me. >> > > I never installed the PV drivers - I read about possible issues it can > cause and figured I can look into it later, if I ever get everything else > working stably. > > Gordan > > > ______________________________**_________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On Tue, 30 Apr 2013 07:54:05 -0400, Casey DeLorme <cdelorme@gmail.com> wrote:> I have been using version 0.11.0.357 for three months on Windows 8 > with passthrough. > > It makes a significant difference in disk (SSD) performance, but I am > not sure about network. > > As Gordan said, not exactly the best choice for stability yet.My VM disk is an iSCSI share, so I''m not too concerned about making latency any less bad at the moment. Just having things work would be nice for now. :)
Ole Johan Væringstad
2013-Apr-30 18:27 UTC
Re: VGA/PCI Passthrough of Secondary Graphics Adapter
I too got the atikmpag.sys BSOD initially, I got around it by installing the 12.104 drivers. It also took me a while to realize I had a "working" solution: I would install the drivers, then reboot the dom0, start the domU, connect vnc, and then windows would hang on "Starting Windows". Turns out windows completes the login (I set up auto-login) on the output of the card I passed through. In device manager the emulated VGA adapter shows an error, while the ATI card is fine. If I reboot domU, the emulated VGA is fine, while the ATI card shows an error. For a while I thought I had to do a dom0 reboot, which fixes it, but ejecting the card also works (although I will have to find a way to shut down the domU "in the dark" which is cleaner than xl destroy). My passthrough card is an Asus HD7970. I initially had an Asus HD7770 as my dom0 card, but that caused problems. I have the 7970 in the first PCIe slot and the dom0 in the second, but I can set in bios which prints POST. I could not get an ATI driver to work in dom0. I tried the open source Radeon driver, which needed mesa 9.1+ for acceleration with Southern Islands/HD7xxx, which again needed a patched llvm I had to pull from the git-repo of what I actually believe is an AMD engineer. Still, glxgears gave me 5-7 fps. It might have worked nicely without acceleration but I needed that for monitor rotation. Note that you only need mesa 9.1+ for SI/HD7xxx. The AMD''s official driver gave me an amputated xorg.log and ugly segfaults. I pulled out the card and installed my old nVidia GTX275, which now works like a charm with the binary driver. I will order something like an nVidia GTX650 because my 275 only supports 2 monitors and I need 3. I got the HD7770 as a dom0 card because I thought it would simplify matters, turns out the opposite was true. But I do not know if this is related to xen, could be a lot of things. I hope this information will be useful to someone beyond myself, because I had to spend quite a bit of time to get it to work. Time is money too, so there comes a point where buying working solutions becomes cheap. I would like to see a database of working hardware combinations. (Atleast I would have loved to when I was setting it up) Setup: Gigabyte GA-X79S-UP5 with i7-3820 Gentoo dom0 at 3.7.10, 3.8.10, will now test 3.9.0 Xen 4.2.0 from portage, xl toolstack nVidia GTX275 as dom0 VGA Asus Radeon HD7770 DCU-II as domU Secondary Passthrough Windows 7 64bit with PV drivers. AMD 12.104 drivers xen kernel options: dom0_mem=8192M,max:8192M dom0_max_vpcus=4 dom0_vcpus_pin iommu=1 xsave=1 linux kernel options: xen-pciback.permissive xen-pciback.hide=(USBbus and VGA card) I anyone have a good solution for cleanly shutting down windows without a monitor, or would like to see more configuration, please give me a shout. - OJ On Tue, Apr 30, 2013 at 2:39 PM, Gordan Bobic <gordan@bobich.net> wrote:> On Tue, 30 Apr 2013 07:54:05 -0400, Casey DeLorme <cdelorme@gmail.com> > wrote: > >> I have been using version 0.11.0.357 for three months on Windows 8 >> with passthrough. >> >> It makes a significant difference in disk (SSD) performance, but I am >> not sure about network. >> >> As Gordan said, not exactly the best choice for stability yet. >> > > My VM disk is an iSCSI share, so I''m not too concerned about making > latency any less bad at the moment. Just having things work would > be nice for now. :) > > > ______________________________**_________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On 04/30/2013 07:27 PM, Ole Johan Væringstad wrote:> I too got the atikmpag.sys BSOD initially, I got around it by installing > the 12.104 drivers. It also took me a while to realize I had a "working" > solution: > > I would install the drivers, then reboot the dom0, start the domU, > connect vnc, and then windows would hang on "Starting Windows". Turns > out windows completes the login (I set up auto-login) on the output of > the card I passed through. In device manager the emulated VGA adapter > shows an error, while the ATI card is fine. If I reboot domU, the > emulated VGA is fine, while the ATI card shows an error. For a while I > thought I had to do a dom0 reboot, which fixes it, but ejecting the card > also works (although I will have to find a way to shut down the domU "in > the dark" which is cleaner than xl destroy).I haven''t found any correlation between dom0 reboots and domU working. 2 days ago, I was rebooting domU multiple times without rebooting dom0. Yesterday it didn''t matter how many times I rebooted either, domU just wouldn''t start up and BSOD-ed every time before the login screen appeared.> My passthrough card is an Asus HD7970. I initially had an Asus HD7770 as > my dom0 card, but that caused problems. I have the 7970 in the first > PCIe slot and the dom0 in the second, but I can set in bios which prints > POST. I could not get an ATI driver to work in dom0. I tried the open > source Radeon driver, which needed mesa 9.1+ for acceleration with > Southern Islands/HD7xxx, which again needed a patched llvm I had to pull > from the git-repo of what I actually believe is an AMD engineer. Still, > glxgears gave me 5-7 fps. It might have worked nicely without > acceleration but I needed that for monitor rotation. Note that you only > need mesa 9.1+ for SI/HD7xxx. The AMD''s official driver gave me an > amputated xorg.log and ugly segfaults. I pulled out the card and > installed my old nVidia GTX275, which now works like a charm with the > binary driver. I will order something like an nVidia GTX650 because my > 275 only supports 2 monitors and I need 3. I got the HD7770 as a dom0 > card because I thought it would simplify matters, turns out the opposite > was true. But I do not know if this is related to xen, could be a lot of > things.My experience is that ATI cards rarely simplify matters. The moment you go off the straight and narrow (single monitor, nothing weird like virtualization) things start to fall apart very quickly, especially in Windows. FGLRX driver is actually pretty decent in Linux, but it''s lack of ability to build for dom0 is a major failing, and likely an unacceptable one for people on this list.> I hope this information will be useful to someone beyond myself, because > I had to spend quite a bit of time to get it to work. Time is money too, > so there comes a point where buying working solutions becomes cheap. I > would like to see a database of working hardware combinations. (Atleast > I would have loved to when I was setting it up) > > Setup: > Gigabyte GA-X79S-UP5 with i7-3820 > Gentoo dom0 at 3.7.10, 3.8.10, will now test 3.9.0 > Xen 4.2.0 from portage, xl toolstack > nVidia GTX275 as dom0 VGA > Asus Radeon HD7770 DCU-II as domU Secondary Passthrough > Windows 7 64bit with PV drivers. > AMD 12.104 drivers > xen kernel options: > dom0_mem=8192M,max:8192M > dom0_max_vpcus=4 > dom0_vcpus_pin > iommu=1 > xsave=1 > linux kernel options: > xen-pciback.permissive > xen-pciback.hide=(USBbus and VGA card)What does xen-pciback.permissive do? I found that regardless of xen-pciback module options, I have to manually detach the devices from dom0 after loading the xen-pciback module. Gordan
> I hope this information will be useful to someone beyond myself, because >> I had to spend quite a bit of time to get it to work. Time is money too, >> so there comes a point where buying working solutions becomes cheap. I >> would like to see a database of working hardware combinations. (Atleast >> I would have loved to when I was setting it up) >> >> Setup: >> Gigabyte GA-X79S-UP5 with i7-3820 >> Gentoo dom0 at 3.7.10, 3.8.10, will now test 3.9.0 >> Xen 4.2.0 from portage, xl toolstack >> nVidia GTX275 as dom0 VGA >> Asus Radeon HD7770 DCU-II as domU Secondary Passthrough >> Windows 7 64bit with PV drivers. >> AMD 12.104 drivers >> xen kernel options: >> dom0_mem=8192M,max:8192M >> dom0_max_vpcus=4 >> dom0_vcpus_pin >> iommu=1 >> xsave=1 >> linux kernel options: >> xen-pciback.permissive >> xen-pciback.hide=(USBbus and VGA card) >> > > What does xen-pciback.permissive do? > > I found that regardless of xen-pciback module options, I have to manually > detach the devices from dom0 after loading the xen-pciback module. > >http://wiki.xen.org/wiki/Xen_PCI_Passthrough#PV_guests_and_PCI_quirks I have never seen this message myself, but apparently without permissive some devices won''t be passed or won''t work as intended. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On Apr 30, 2013, at 8:41 AM, "Gordan Bobic [via Xen]" <ml-node+s1045712n5715774h45@n5.nabble.com> wrote:> On 04/30/2013 07:27 PM, Ole Johan Væringstad wrote: > > > I too got the atikmpag.sys BSOD initially, I got around it by installing > > the 12.104 drivers. It also took me a while to realize I had a "working" > > solution: > > > > I would install the drivers, then reboot the dom0, start the domU, > > connect vnc, and then windows would hang on "Starting Windows". Turns > > out windows completes the login (I set up auto-login) on the output of > > the card I passed through. In device manager the emulated VGA adapter > > shows an error, while the ATI card is fine. If I reboot domU, the > > emulated VGA is fine, while the ATI card shows an error. For a while I > > thought I had to do a dom0 reboot, which fixes it, but ejecting the card > > also works (although I will have to find a way to shut down the domU "in > > the dark" which is cleaner than xl destroy). > > I haven''t found any correlation between dom0 reboots and domU working. 2 > days ago, I was rebooting domU multiple times without rebooting dom0. > Yesterday it didn''t matter how many times I rebooted either, domU just > wouldn''t start up and BSOD-ed every time before the login screen appeared. > > > My passthrough card is an Asus HD7970. I initially had an Asus HD7770 as > > my dom0 card, but that caused problems. I have the 7970 in the first > > PCIe slot and the dom0 in the second, but I can set in bios which prints > > POST. I could not get an ATI driver to work in dom0. I tried the open > > source Radeon driver, which needed mesa 9.1+ for acceleration with > > Southern Islands/HD7xxx, which again needed a patched llvm I had to pull > > from the git-repo of what I actually believe is an AMD engineer. Still, > > glxgears gave me 5-7 fps. It might have worked nicely without > > acceleration but I needed that for monitor rotation. Note that you only > > need mesa 9.1+ for SI/HD7xxx. The AMD''s official driver gave me an > > amputated xorg.log and ugly segfaults. I pulled out the card and > > installed my old nVidia GTX275, which now works like a charm with the > > binary driver. I will order something like an nVidia GTX650 because my > > 275 only supports 2 monitors and I need 3. I got the HD7770 as a dom0 > > card because I thought it would simplify matters, turns out the opposite > > was true. But I do not know if this is related to xen, could be a lot of > > things. > > My experience is that ATI cards rarely simplify matters. The moment you > go off the straight and narrow (single monitor, nothing weird like > virtualization) things start to fall apart very quickly, especially in > Windows. FGLRX driver is actually pretty decent in Linux, but it''s lack > of ability to build for dom0 is a major failing, and likely an > unacceptable one for people on this list. > > > I hope this information will be useful to someone beyond myself, because > > I had to spend quite a bit of time to get it to work. Time is money too, > > so there comes a point where buying working solutions becomes cheap. I > > would like to see a database of working hardware combinations. (Atleast > > I would have loved to when I was setting it up) > > > > Setup: > > Gigabyte GA-X79S-UP5 with i7-3820 > > Gentoo dom0 at 3.7.10, 3.8.10, will now test 3.9.0 > > Xen 4.2.0 from portage, xl toolstack > > nVidia GTX275 as dom0 VGA > > Asus Radeon HD7770 DCU-II as domU Secondary Passthrough > > Windows 7 64bit with PV drivers. > > AMD 12.104 drivers > > xen kernel options: > > dom0_mem=8192M,max:8192M > > dom0_max_vpcus=4 > > dom0_vcpus_pin > > iommu=1 > > xsave=1 > > linux kernel options: > > xen-pciback.permissive > > xen-pciback.hide=(USBbus and VGA card) > > What does xen-pciback.permissive do? > > I found that regardless of xen-pciback module options, I have to > manually detach the devices from dom0 after loading the xen-pciback module.I found that using pciback as a module would result in similar behavior. Since my graphics cards are both AMD Radeons, the open-source ''radeon'' driver would load with the kernel due to the new requirement for kernel-mode switching. The FGLRX drivers do not do this, and can be disabled with pciback as a module. Since I couldn''t'' get the FGLRX drivers to work with the Xen kernel (in Fedora, w/3.8 kernel, whereas in Ubuntu 12.04 with 3.5 it worked fine), I recompiled 3.8 with pciback statically linked. This gave me the ability to hide the device immediately upon kernel loading and prevents the radeon driver from initializing the device. However, I never got a display on my monitor; but I think I will the methods posted in this thread (regarding rebooting at certain points). I think they may help, and also explain some of the erratic behavior I''ve witnessed. I just wish this were a little more straightforward, and I question the long-term reliability of using PCI passthrough with Radeon cards. I''m assuming the Quadro cards work flawlessly? -- View this message in context: http://xen.1045712.n5.nabble.com/VGA-PCI-Passthrough-of-Secondary-Graphics-Adapter-tp5715723p5715775.html Sent from the Xen - User mailing list archive at Nabble.com. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
>If only ATI provided a method to contact them for such support,considering they claim that they "tested" 4850/4870 cards to work with Xen and VGA >passthrough. If they''ve tested it, it''d be nice if they documented it. Personally I''ve not seen any evidence of the sort of stability you''d rightfully expect when >the manufacturer vouches that something works. Which toolstack are you using? I have had better success with xm than with xl for passing through a 6850 to a windows 7 domU. Namely, I experienced the performance degradation on reboot of domU with xl, but not with xm. Though I could solve it also under xl by the eject trick after reboot of domU - but if this is a general reset related issue it might be different symptoms for different radeon models. This was in xen 4.1.x under gentoo and ubuntu (haven''t tested xen 4.2+). Relying on a deprecated toolstack is not future-proof of course, but it has worked for me so far, and ubuntu 12.10 still uses xen 4.1 which provides xm as the default. It might be that AMD got their tested cards working with xen 4.1 and xm, and never tried any more after that. Regards, Andreas _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On 05/01/2013 08:42 AM, Andreas Falck wrote:> >If only ATI provided a method to contact them for such support, > considering they claim that they "tested" 4850/4870 cards to work with > Xen and VGA >passthrough. If they''ve tested it, it''d be nice if they > documented it. Personally I''ve not seen any evidence of the sort of > stability you''d rightfully expect when >the manufacturer vouches that > something works. > > Which toolstack are you using? I have had better success with xm than > with xl for passing through a 6850 to a windows 7 domU. Namely, I > experienced the performance degradation on reboot of domU with xl, but > not with xm.I am using the xm stack.> Though I could solve it also under xl by the eject trick > after reboot of domU - but if this is a general reset related issue it > might be different symptoms for different radeon models. This was in xen > 4.1.x under gentoo and ubuntu (haven''t tested xen 4.2+). Relying on a > deprecated toolstack is not future-proof of course, but it has worked > for me so far, and ubuntu 12.10 still uses xen 4.1 which provides xm as > the default. > > It might be that AMD got their tested cards working with xen 4.1 and xm, > and never tried any more after that.I''m taking the claims of AMD having "tested" things with a bucket of salt until I see the detailed documentation of the process they used (exact Xen version, dom0 OS stack, domU version and patch level, and the exact tests carried out in domU). Thus far I have seen no such documentation, which makes the claims that it was tested pure hearsay. Gordan
2013/4/29 Gordan Bobic <gordan@bobich.net>> > Also, does anyone have an ATI FirePro card that they could post lspci > output from? I can believe that those might have FLreset+ to compensate for > the driver not being up to the task. > >lspci -vv from my v4900 below. It looks like it doesn''t have FLR. I have successfully passed it through to a windows 8 domU once, just for testing, with seemingly good results. This was with ubuntu 12.10 with vanilla everything (xen 4.1.3, xm toolstack - my 6850 worked flawlessly with passthrough in the same setup). Regards, Andreas>>05:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Turks [FirePro V4900] (prog-if 00 [VGA controller]) Subsystem: Dell Device 240a Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 154 Region 0: Memory at c0000000 (64-bit, prefetchable) [size=256M] Region 2: Memory at dffe0000 (64-bit, non-prefetchable) [size=128K] Region 4: I/O ports at d000 [size=256] Expansion ROM at dffc0000 [disabled] [size=128K] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee20000 Data: 4042 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Kernel driver in use: radeon Kernel modules: radeon _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Ok, I just started testing a radeon 7790 with my setup, and it seems I get reset issues after reboot now with xm. No BSOD but the card shows up as unusable (yellow exlamation mark) after reboot of domU. I''ll update as testing progresses. Regards, Andreas 2013/5/1 Gordan Bobic <gordan@bobich.net>> On 05/01/2013 08:42 AM, Andreas Falck wrote: > >> >If only ATI provided a method to contact them for such support, >> considering they claim that they "tested" 4850/4870 cards to work with >> Xen and VGA >passthrough. If they''ve tested it, it''d be nice if they >> documented it. Personally I''ve not seen any evidence of the sort of >> stability you''d rightfully expect when >the manufacturer vouches that >> something works. >> >> Which toolstack are you using? I have had better success with xm than >> with xl for passing through a 6850 to a windows 7 domU. Namely, I >> experienced the performance degradation on reboot of domU with xl, but >> not with xm. >> > > I am using the xm stack. > > > Though I could solve it also under xl by the eject trick >> after reboot of domU - but if this is a general reset related issue it >> might be different symptoms for different radeon models. This was in xen >> 4.1.x under gentoo and ubuntu (haven''t tested xen 4.2+). Relying on a >> deprecated toolstack is not future-proof of course, but it has worked >> for me so far, and ubuntu 12.10 still uses xen 4.1 which provides xm as >> the default. >> >> It might be that AMD got their tested cards working with xen 4.1 and xm, >> and never tried any more after that. >> > > I''m taking the claims of AMD having "tested" things with a bucket of salt > until I see the detailed documentation of the process they used (exact Xen > version, dom0 OS stack, domU version and patch level, and the exact tests > carried out in domU). Thus far I have seen no such documentation, which > makes the claims that it was tested pure hearsay. > > > Gordan > > ______________________________**_________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Does the FirePro exhibit the same issue? Or does it "just work"? On Wed, 1 May 2013 10:10:05 +0200, Andreas Falck <falck.andreas.lists@gmail.com> wrote:> Ok, I just started testing a radeon 7790 with my setup, and it seems > I > get reset issues after reboot now with xm. No BSOD but the card shows > up as unusable (yellow exlamation mark) after reboot of domU. I'll > update as testing progresses. > > Regards, > Andreas > > 2013/5/1 Gordan Bobic > > On 05/01/2013 08:42 AM, Andreas Falck wrote: > >If only ATI provided a method to contact them for such support, > considering they claim that they "tested" 4850/4870 cards to work > with > Xen and VGA >passthrough. If they've tested it, it'd be nice if they > documented it. Personally I've not seen any evidence of the sort of > stability you'd rightfully expect when >the manufacturer vouches > that > something works. > > Which toolstack are you using? I have had better success with xm > than > with xl for passing through a 6850 to a windows 7 domU. Namely, I > experienced the performance degradation on reboot of domU with xl, > but > not with xm. > > I am using the xm stack. > > Though I could solve it also under xl by the eject trick > after reboot of domU - but if this is a general reset related issue > it > might be different symptoms for different radeon models. This was in > xen > 4.1.x under gentoo and ubuntu (haven't tested xen 4.2+). Relying on > a > deprecated toolstack is not future-proof of course, but it has > worked > for me so far, and ubuntu 12.10 still uses xen 4.1 which provides xm > as > the default. > > It might be that AMD got their tested cards working with xen 4.1 and > xm, > and never tried any more after that. > > I'm taking the claims of AMD having "tested" things with a bucket of > salt until I see the detailed documentation of the process they used > (exact Xen version, dom0 OS stack, domU version and patch level, and > the exact tests carried out in domU). Thus far I have seen no such > documentation, which makes the claims that it was tested pure > hearsay. > > Gordan > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org [2] > http://lists.xen.org/xen-users [3] > > > > Links: > ------ > [1] mailto:gordan@bobich.net > [2] mailto:Xen-users@lists.xen.org > [3] http://lists.xen.org/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
2013/5/1 Gordan Bobic <gordan@bobich.net>> Does the FirePro exhibit the same issue? Or does it "just work"? > >I realize when I think about it that I am unsure if I ever tried rebooting the domU with the firepro in it. I typically would have tried, but I''m not sure. I may add that I used the driver bundled with windows 8 for both the 6850 (which worked with rebooting) and for the firepro, but for the 7790 I had to install catalyst (since the card is newer than win8). Regards, Andreas _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users