Hello Xen Users,
I have attached the latest copy of my own documentation in PDF format as
well as some error files related to the latest tests I ran. I have
documented the steps I took to reach my current situation. While I have a
"working" install, it is not optimal and there are several problems
and
questions I would like to address:
- If Dom0 is a privileged DomU, should we be enabling the DomU kernel flags
in addition to the Dom0 kernel flags?
- What purposes does xen-watchdog server, is it only for xend, and do we
still need it for xl?
- What benefits can be had by limiting Dom0 vcpu''s in grub?
- Upstream qemu fails to load virtual machines with VGA passthrough and a
large amount of memory (3600MB+ in my case breaks the DomU).
- Does anyone know exactly what Windows device ejection does to the
hardware, or how we can do the same from Linux (such as Dom0)?
I would appreciate any suggestions or answers that you can provide to these
questions.
---
**A note on GPLPV:**
Using the latest GPLPV, and so far it works excellent. To be honest I
don''t notice a different with regards to disk IO, solid state is
already
fast, but the Windows Experience index jumps from a 6.6 to a 7.9.
What is really astounding is network traffic. I run a DomU Samba File
Server and without GPLPV I manage 50~Mbps for file transfers. With GLPV
that jumps to an average over 200~Mbps.
---
**A note on passthrough and upstream qemu:**
I tried followed David Techer''s latest blog post, but I don''t
use the xm
toolstack or qemu-traditional. I only wanted the memory patch for
passthrough with upstream, since I am still limited to 3GB.
I followed these steps to build:
git clone git://xenbits.xen.org/xen.git
cd xen
git checkout -b stable-4.3 origin/stable-4.3
cd tools
make -j9 && make clean
cd qemu-xen-dir
wget
"http://marc.info/?l=qemu-devel&m=136177475215360&q=raw" -O -
|
patch -p1
cd ../..
make -j9 world && make -j9 debball
cd dist
dpkg -i *.deb
The patch applied successfully, and xen built and installed without a
problem. However, when I attempted to boot my Windows DomU (no changes to
the config yet just to see if it would boot), it failed. I have attached
the output, as well as the BSoD screen Windows displayed over VNC.
Without the patch process (lines `cd tools` to `cd ../..`), I can boot my
Windows DomU but I cannot supply large amounts of memory.
---
Testing sysfs reset:
Modern linux kernel sysfs comes with reset files that can be used to reset
(some) PCI devices manually:
- [Kernel Docs (
https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci)](https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci
)
I decided to give this a try to see if it would allow me to reset the
adapter from within Linux, where I could then tie a script to automate the
reset process when a DomU is rebooted.
The planned scenario:
- Windows boots and initializes the graphics card
- I shut down windows and the card remains initialized
- I reset the graphics card state by:
- Unbinding from pciback
- issuing a reset
- rebinding it
- Booting windows should initialize a fresh card
I decided to try the following set of commands and reboot my Windows DomU
between each to see if performance was still a concern (duplicate commands
for audio device omitted):
First I tried simply unbinding and rebinding to pciback:
echo 0000:03:00.0 > /sys/bus/pci/devices/0000\:03\:00.0/driver/unbind
echo 0000:03:00.0 > /sys/bus/pci/drivers/pciback/new_slot
echo 0000:03:00.0 > /sys/bus/pci/drivers/pciback/bind
This did nothing so I tried unbinding, resetting, and rebinding:
echo 0000:03:00.0 > /sys/bus/pci/devices/0000\:03\:00.0/driver/unbind
echo 1 > /sys/bus/pci/devices/0000\:03\:00.0/reset
echo 0000:03:00.0 > /sys/bus/pci/drivers/pciback/new_slot
echo 0000:03:00.0 > /sys/bus/pci/drivers/pciback/bind
This did not work, and also threw an error when I attempred to reset the
device. From various tests I found that the reset would fail if the card
was not bound to a driver or enable contained a 0. leaving it bound or
echoing a 1 into the enable file before issuing a reset still did not
resolve the degradation.
Some possible conclusions I drew from these failures:
- I am to believe that Windows ejection is probably working because it is
using AMD drivers.
- The reset in Linux fails when it has no drivers so the reset probably
triggers a driver operation
- The driver operation probably fails because it is not tied to an AMD
driver
Another option I have not yet tested would be loading the radon driver to
bind and unbind it before adding it back to pciback, which may cause the
proper reset chain to occur. I didn''t see it in the drivers list
though
and wouldn''t know where to begin loading it without causing problems.
If anyone knows how to cause a D0>D3>D0 power change to a device through
sysfs let me know because I would like to try that next.
Thank you for your time,
~Casey
_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users
On Sun, 14 Jul 2013 17:58:46 -0400, Casey DeLorme <cdelorme@gmail.com> wrote:> - Upstream qemu fails to load virtual machines with VGA passthrough > and a large amount of memory (3600MB+ in my case breaks the DomU).I may be wrong, but my understanding is that the PCI passthrough related BAR memory patch was for qemu-traditional, not upstream.> - Does anyone know exactly what Windows device ejection does to the > hardware, or how we can do the same from Linux (such as Dom0)?I suspect it does "whatever the driver does", rather than something defined by a standard of some sort. FWIW, ejecting a device only ever even succeeded for me on Win7. If I try to eject a GPU in XP, it refuses to do so because the "device is busy".> **A note on GPLPV:** > > Using the latest GPLPV, and so far it works excellent. To be honest > I don't notice a different with regards to disk IO, solid state is > already fast, but the Windows Experience index jumps from a 6.6 to a > 7.9.Really? I found the difference is _enormous_. Booting domU takes seconds rather than minutes, and running any kind of anti-virus grinds the machine to a halt without PV disk drivers.> Testing sysfs reset: > > Modern linux kernel sysfs comes with reset files that can be used to > reset (some) PCI devices manually: > > - [Kernel Docs > > (https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci)](https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci > [3]) > > I decided to give this a try to see if it would allow me to reset the > adapter from within Linux, where I could then tie a script to > automate > the reset process when a DomU is rebooted. > > The planned scenario: > > - Windows boots and initializes the graphics card > - I shut down windows and the card remains initialized > - I reset the graphics card state by: > - Unbinding from pciback > - issuing a reset > - rebinding it > - Booting windows should initialize a fresh cardI think you'll find this process is entirely at the mercy of what the driver does in domU. Quadro drivers seem to handle this very gracefully. Primary passthrough might work better because it re-executes the BIOS which may well get the card to a clean state, but I am purely guessing since I have given up on ATI cards some time ago for number of reasons.> - I am to believe that Windows ejection is probably working because > it > is using AMD drivers.Ejecting a Quadro card on Win7 "worked" for me, but I never actually saw any benefit from doing so with Quadro cards since they work fine after a domU reboot anyway.> - The reset in Linux fails when it has no drivers so the reset > probably triggers a driver operationYou have a reset option under /sys/ when the driver is loaded? I've never seen that. I thought it was specifically related to FLreset PCI level functionality.> - The driver operation probably fails because it is not tied to an > AMD > driverAnd you have definitely confirmed that it does something (or even exists) when the fglrx driver is claiming the device?> Another option I have not yet tested would be loading the radon > driver > to bind and unbind it before adding it back to pciback, which may > cause the proper reset chain to occur. I didn't see it in the > drivers list though and wouldn't know where to begin loading it > without causing problems.Well, you can modprobe fglrx and see if/what it breaks. :)> If anyone knows how to cause a D0>D3>D0 power change to a device > through sysfs let me know because I would like to try that next.Hmm... Abusing power management - I like the idea. :) It is not likely to work if the card takes auxiliary power input, though. :( Gordan _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Thanks for the reply Gordon. On Mon, Jul 15, 2013 at 11:02 AM, Gordan Bobic <gordan@bobich.net> wrote:> On Sun, 14 Jul 2013 17:58:46 -0400, Casey DeLorme <cdelorme@gmail.com> > wrote: > > - Upstream qemu fails to load virtual machines with VGA passthrough >> and a large amount of memory (3600MB+ in my case breaks the DomU). >> > > I may be wrong, but my understanding is that the PCI passthrough > related BAR memory patch was for qemu-traditional, not upstream. > > The patch points to files in the qemu-xen-dir notqemu-xen-traditional-dir, so I am pretty sure it is for upstream. If it wasn''t then it should have had no effect when added, instead it broke my HVM using upstream-qemu.> > - Does anyone know exactly what Windows device ejection does to the >> hardware, or how we can do the same from Linux (such as Dom0)? >> > > I suspect it does "whatever the driver does", rather than something > defined by a standard of some sort. > > FWIW, ejecting a device only ever even succeeded for me on Win7. > If I try to eject a GPU in XP, it refuses to do so because the > "device is busy". > > I had ejections working fine with 4.2 and Windows 8. However,upstream-qemu provides way smoother performance for a number of things, so ideally I would like to use it instead of traditional.> > **A note on GPLPV:** >> >> Using the latest GPLPV, and so far it works excellent. To be honest >> I don''t notice a different with regards to disk IO, solid state is >> already fast, but the Windows Experience index jumps from a 6.6 to a >> 7.9. >> > > Really? I found the difference is _enormous_. Booting domU takes > seconds rather than minutes, and running any kind of anti-virus > grinds the machine to a halt without PV disk drivers. > > Actually this is exactly one of the things upstream qemu addressed,Windows 8 boot time on 4.2 traditional was upwards of 2 minutes even with GPLPV installed. GPLPV made almost no difference visible for my boot times, software may or may not be runnuing faster. I''m sure it is, but I don''t notice the difference. SSD''s are fast, so between fast and faster the line gets blurry I guess. If I was using an HDD it would probably be a different story. Testing sysfs reset:>> >> Modern linux kernel sysfs comes with reset files that can be used to >> reset (some) PCI devices manually: >> >> - [Kernel Docs >> >> (https://www.kernel.org/doc/**Documentation/ABI/testing/** >> sysfs-bus-pci)](https://www.**kernel.org/doc/Documentation/** >> ABI/testing/sysfs-bus-pci<https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci)%5D(https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci> >> [3]) >> >> >> I decided to give this a try to see if it would allow me to reset the >> adapter from within Linux, where I could then tie a script to automate >> the reset process when a DomU is rebooted. >> >> The planned scenario: >> >> - Windows boots and initializes the graphics card >> - I shut down windows and the card remains initialized >> - I reset the graphics card state by: >> - Unbinding from pciback >> - issuing a reset >> - rebinding it >> - Booting windows should initialize a fresh card >> > > I think you''ll find this process is entirely at the mercy of > what the driver does in domU. Quadro drivers seem to handle > this very gracefully. > > This is good news, because I am hoping Linux handles things the same way.> Primary passthrough might work better because it re-executes > the BIOS which may well get the card to a clean state, but > I am purely guessing since I have given up on ATI cards > some time ago for number of reasons. > > That is possible, I never had luck getting primary passthrough workingbefore, maybe I will try again. However then I have to use traditional qemu again, so again ideally I''d rather use upstream and work around secondary passthrough.> - I am to believe that Windows ejection is probably working because it >> is using AMD drivers. >> > > Ejecting a Quadro card on Win7 "worked" for me, but I never > actually saw any benefit from doing so with Quadro cards > since they work fine after a domU reboot anyway. > > If I could achieve that with an AMD I would be happy, but I haven''t foundany good instructions on how to actually mod the GTX to Quadro that doesn''t involve hardware modifications.> > - The reset in Linux fails when it has no drivers so the reset >> probably triggers a driver operation >> > > You have a reset option under /sys/ when the driver is loaded? > I''ve never seen that. I thought it was specifically related > to FLreset PCI level functionality. > > Supposedly the reset files were an alternative addition to the `do_flr`?I did read a little bit about it, but not much by way of documentation around it yet.> > - The driver operation probably fails because it is not tied to an AMD >> driver >> > > And you have definitely confirmed that it does something (or even > exists) when the fglrx driver is claiming the device? > > I have not, but if it''s anything like Windows then this is exactly whatshould be happening right? I am basing this off of that thought and the fact that if no driver is attached the reset throws an error. It''s all speculation right now, I was hoping someone with knowledge about pciback or sysfs could confirm it. If that is the case there is a strong possibility attaching it to say the radeon or fglrx driver would handle a reset properly. I did test resetting emulated graphics in a virtual machine successfully, so I can say that the reset appears to do "something".> Another option I have not yet tested would be loading the radon driver >> to bind and unbind it before adding it back to pciback, which may >> cause the proper reset chain to occur. I didn''t see it in the >> drivers list though and wouldn''t know where to begin loading it >> without causing problems. >> > > Well, you can modprobe fglrx and see if/what it breaks. :) > > Good idea, I will have to install fglrx first, but hopefully that willload the driver into `/sys/bus/pci/drivers`.> If anyone knows how to cause a D0>D3>D0 power change to a device >> through sysfs let me know because I would like to try that next. >> > > Hmm... Abusing power management - I like the idea. :) > It is not likely to work if the card takes auxiliary power > input, though. :( > > Hmm good point, it does take two auxiliary power inputs. I thought D0/D3operations were for device hibernation, does auxiliary power prevent that from being possible?> Gordan >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On Mon, 15 Jul 2013 11:33:32 -0400, Casey DeLorme <cdelorme@gmail.com> wrote:> - Upstream qemu fails to load virtual machines with VGA passthrough > and a large amount of memory (3600MB+ in my case breaks the DomU). > > I may be wrong, but my understanding is that the PCI passthrough > related BAR memory patch was for qemu-traditional, not upstream. > > The patch points to files in the qemu-xen-dir not > qemu-xen-traditional-dir, so I am pretty sure it is for upstream. If > it wasn't then it should have had no effect when added, instead it > broke my HVM using upstream-qemu.Don't quote me on this (because I don't have time right now to search through the xen-devel archives), but IIRC no changes were made to qemu-upstream because it was deemed to not suffer from the PCI memory overrun issue. But either way, if it broke something, that's bad. You should probably fire off a report on the -devel mailing list about it.> SSD's are fast, so between fast and faster the line gets blurry I > guess. If I was using an HDD it would probably be a different story.It's the other way around - you'll notice _more_ difference on an SSD because the overhead will be much greater compared to what the disk is capable of. Without PV disk drivers my bottleneck was qemu-dm running out of CPU in dom0.> Testing sysfs reset: > > Modern linux kernel sysfs comes with reset files that can be used to > reset (some) PCI devices manually: > > - [Kernel Docs > > > > (https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci)](https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci > [3] > [3]) > > I decided to give this a try to see if it would allow me to reset > the > adapter from within Linux, where I could then tie a script to > automate > the reset process when a DomU is rebooted. > > The planned scenario: > > - Windows boots and initializes the graphics card > - I shut down windows and the card remains initialized > - I reset the graphics card state by: > - Unbinding from pciback > - issuing a reset > - rebinding it > - Booting windows should initialize a fresh card > > I think you'll find this process is entirely at the mercy of > what the driver does in domU. Quadro drivers seem to handle > this very gracefully. > > This is good news, because I am hoping Linux handles things the same > way.That would be good news if we knew exactly what the Nvidia driver does in domU. But we don't, and are unlikely to ever find out. As I mentioned before, I gave up on ATI cards for more than just this reason, and am happily using Quadrified (and soon Gridified) GeForce cards for my requirements.> Primary passthrough might work better because it re-executes > the BIOS which may well get the card to a clean state, but > I am purely guessing since I have given up on ATI cards > some time ago for number of reasons. > > That is possible, I never had luck getting primary passthrough > working > before, maybe I will try again. However then I have to use > traditional qemu again, so again ideally I'd rather use upstream and > work around secondary passthrough.I hate to say this, but my experience is that if you are after an easy option - using ATI cards isn't it. Don't get me wrong - I still use an ATI 4850 card for dom0 on my system; but that is only because: 1) I cannot get anything but ATI cards to work in slot 1 on my EVGA SR-2 (bizzare as that may sound, tried 7 different Nvidia cards, 1 sound card, 2 disk controllers and 2 ATI cards - and of all those only the 2 ATI cards worked in that slot) 2) 4850 because Radeon 4xxx is the last series that had two dual-link DVI ports (all later cards only have single link wired into the 2nd DVI port) 3) It means I can just blacklist nvidia in dom0 and not have to selectively work around the Nvidia GPU I use for dom0. 4) It is the best available within the above constraints in single-slot form factor. Using the right tool for the job helps. :)> - I am to believe that Windows ejection is probably working because > it > is using AMD drivers. > > Ejecting a Quadro card on Win7 "worked" for me, but I never > actually saw any benefit from doing so with Quadro cards > since they work fine after a domU reboot anyway. > > If I could achieve that with an AMD I would be happy, but I haven't > found any good instructions on how to actually mod the GTX to Quadro > that doesn't involve hardware modifications.I'm in the process of writing up a detailed article on this at the moment, but 5xx and earlier series cards only require soft-modding. The only 5xx (still soft-moddable) series card worth considering is the 580 which is modifiable into a Quadro 7000 - but it's pointless because 7000 isn't "MultiOS" capable, so the Quadro driver doesn't do it's magic to make it work in domU. 6xx series cards do require a small hardware modification in addition to BIOS modification, and to do the BIOS modification properly you have to strip out all of the UEFI crap they have put in there (also doable - not tested my method yet, but will do so when time permits). FYI, I'm currently running modified GTS450 (Q2000), GTX470 (Q5000) and GTX480 (Q6000) cards.> - The reset in Linux fails when it has no drivers so the reset > probably triggers a driver operation > > You have a reset option under /sys/ when the driver is loaded? > I've never seen that. I thought it was specifically related > to FLreset PCI level functionality. > > Supposedly the reset files were an alternative addition to the > `do_flr`? I did read a little bit about it, but not much by way of > documentation around it yet.So presumably that depends on the driver implementing the reset.> - The driver operation probably fails because it is not tied to an > AMD > driver > > And you have definitely confirmed that it does something (or even > exists) when the fglrx driver is claiming the device? > > I have not, but if it's anything like Windows then this is exactly > what should be happening right? I am basing this off of that thought > and the fact that if no driver is attached the reset throws an error. > It's all speculation right now, I was hoping someone with knowledge > about pciback or sysfs could confirm it.I really wouldn't make any assumptions about what the GPU driver might or might not do, especially since my experience, especially in the case of ATI, is that their Windows and Linux drivers are quite fundamentally different.> If that is the case there is a strong possibility attaching it to say > the radeon or fglrx driver would handle a reset properly. > > I did test resetting emulated graphics in a virtual machine > successfully, so I can say that the reset appears to do "something".You are probably better of going the primary passthrough route and getting the BIOS to re-post the card. The only reason why I haven't tried that with my Quadrified cards is because, well, they work as it is. I can live with the loading screen only being available in VNC.> Another option I have not yet tested would be loading the radon > driver > to bind and unbind it before adding it back to pciback, which may > cause the proper reset chain to occur. I didn't see it in the > drivers list though and wouldn't know where to begin loading it > without causing problems. > > Well, you can modprobe fglrx and see if/what it breaks. :) > > Good idea, I will have to install fglrx first, but hopefully that > will > load the driver into `/sys/bus/pci/drivers`.Good luck. :)> If anyone knows how to cause a D0>D3>D0 power change to a device > through sysfs let me know because I would like to try that next. > > Hmm... Abusing power management - I like the idea. :) > It is not likely to work if the card takes auxiliary power > input, though. :( > > Hmm good point, it does take two auxiliary power inputs. I thought > D0/D3 operations were for device hibernation, does auxiliary power > prevent that from being possible?No idea. On the SR-2 I tried toggling PCIe slot enable jumpers to try to reset the card at runtime, but it didn't do anything of the sort. As far as I can tell, it only tells the BIOS to hide the device at POST time. Gordan _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users