Radoslaw Szkodzinski
2012-Jun-24 11:48 UTC
PCI Passthrough, Radeon 7950 and Windows 7 64-bit
Hello, I've been consistently attempting to make the setup mentioned in the subject work on the following hardware: Xeon E3 1275 (v1, Sandy Bridge) Asus P8B WS (Firmware: 0904 and 2009) 16 GB RAM Sapphire Radeon HD 7950 OC (Firmware: 015.013.000.010.000705) Linux 3.4.1 dom0, x86-64, pvops. Xen 4.1.2 (+ security patch) and 4.1.3-rc1. Old qemu-dm setup. (not new mainline qemu) Windows 7 64-bit (actually debug build), 4 GB allocated memory Signed Uninvention GPLPV drivers 0.11.0.356 (in use for disk and networking) I've heard others managed to get this setup running without issues. What I've been able to get is the passthrough to work exactly once per start of the HVM. Second start of the HVM Windows 7 64-bit results in a bluescreen mentioning a page fault. I've also attempted to force a PCI bus reset on the card via sysfs and also a forced detach and rescan - trying to passthrough the card afterwards results in a hard hang. Same result if xen-pciback is set with passthrough=1. I haven't found a way to force a power state reset yet (d3->d0->d3), it'd be useful for a test. Any other ideas how to debug this? I can provide various logs as necessary. [Note: I'm not subscribed to the list, please CC.] -- Radosław Szkodziński _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Radoslaw Szkodzinski
2012-Jun-25 13:50 UTC
PCI Passthrough, Radeon 7950 and Windows 7 64-bit
Hello, I've been consistently attempting to make the setup mentioned in the subject work on the following hardware: Xeon E3 1275 (v1, Sandy Bridge) Asus P8B WS (Firmware: 0904, 2009 and 2102) 16 GB RAM Sapphire Radeon HD 7950 OC (Firmware: 015.013.000.010.000705) Linux 3.4.1 dom0, x86-64, pvops. Xen 4.1.2 (+ security patch) and 4.1.3-rc1. Old qemu-dm setup. (not new mainline qemu) Windows 7 64-bit (actually debug build), 4 GB allocated memory Signed Uninvention GPLPV drivers 0.11.0.356 (in use for disk and networking) I've heard others managed to get this setup running without issues. What I've been able to get is the passthrough to work exactly once per start of the HVM. Second start of the HVM Windows 7 64-bit results in a bluescreen mentioning a page fault. I've also attempted to force a PCI bus reset on the card via sysfs and also a forced detach and rescan - trying to passthrough the card afterwards results in a hard hang. Same result if xen-pciback.passthrough=1. I haven't found a way to force a power state reset yet (d3->d0->d3), it'd be useful for a test. Any other ideas how to debug this? I can provide various logs as necessary. [Sorry about any possible double-post. I've registered to the list.] -- Radosław Szkodziński _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Radoslaw Szkodzinski
2012-Jun-25 15:09 UTC
Re: PCI Passthrough, Radeon 7950 and Windows 7 64-bit
On Mon, Jun 25, 2012 at 4:52 PM, Matthias <matthias.kannenberg@googlemail.com> wrote:> Hi, > > I got a quite similar setup working with vga passthrough. Can you > provide some more information? > > What kernel are you using? With xen patches? > Everything xen related compiled from source or any distro packages?Qubes packages for libs (that's Xen 4.1.2 patched), the other 4.1.3-rc1 hypervisor by myself from Gentoo ebuild.> Are you using gfx_passthrough=1 ?No, secondary PCI card.> qemu-dm or stubdom?Mentioned in the post, qemu-dm. I was unable to make stubdom work with the rest of the networking setup yet.> xl or xm?Xl. No xm here anymore, as xend is not started.> Hiding the vga card on boot and passing it to the domu via config in > pci = [] or doing everything on the fly?Statically, as in pci = [ '0000:01:00.0', '0000:01:00.1' ].> > If this is any help: my setup is running on a debian wheezy with an > openSuse Kernel (3.4.2) and their xen patches. I'm hiding my vga on > boot via pciback.hide (no other pciback params)Here, it's being hidden by the initramfs using sysfs, before modules being loaded. I can attach the script that does this - it just adds the slots and binds the devices to pciback.> For pagefaults, i find the dmesg and xm dmesg outputs more informative > than the actual xen logs for most of the time.. anything suspicious > coming up there?Nothing obvious, that's the problem. I'll attach a whole collection of logs later. Thanks, -- Radosław Szkodziński _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Maybe you should give xm a try just to see if it does the trick. I never got vga passthrough working with xl (and from my understanding, it''s a lot mor complicated there with compiling the vga bios into xen and manual calculating vga adress ranges.. with xm, I''m doing neither of it). Also, do you increase the log level for xen? my kernel line is: multiboot /boot/xen.gz dom0_mem=2048M iommu=1 loglvl=all guest_loglvl=all What kernel are you using? If you want i can provide my build commands for the xen-patched openSuse Kernel.. 2012/6/25 Radoslaw Szkodzinski <astralstorm@gmail.com>:> On Mon, Jun 25, 2012 at 4:52 PM, Matthias > <matthias.kannenberg@googlemail.com> wrote: >> Hi, >> >> I got a quite similar setup working with vga passthrough. Can you >> provide some more information? >> >> What kernel are you using? With xen patches? >> Everything xen related compiled from source or any distro packages? > > Qubes packages for libs (that''s Xen 4.1.2 patched), the other > 4.1.3-rc1 hypervisor by myself from Gentoo ebuild. > >> Are you using gfx_passthrough=1 ? > > No, secondary PCI card. > >> qemu-dm or stubdom? > > Mentioned in the post, qemu-dm. I was unable to make stubdom work with > the rest of the networking setup yet. > >> xl or xm? > > Xl. No xm here anymore, as xend is not started. > >> Hiding the vga card on boot and passing it to the domu via config in >> pci = [] or doing everything on the fly? > > Statically, as in pci = [ ''0000:01:00.0'', ''0000:01:00.1'' ]. > >> >> If this is any help: my setup is running on a debian wheezy with an >> openSuse Kernel (3.4.2) and their xen patches. I''m hiding my vga on >> boot via pciback.hide (no other pciback params) > > Here, it''s being hidden by the initramfs using sysfs, before modules > being loaded. > I can attach the script that does this - it just adds the slots and > binds the devices to pciback. > >> For pagefaults, i find the dmesg and xm dmesg outputs more informative >> than the actual xen logs for most of the time.. anything suspicious >> coming up there? > > Nothing obvious, that''s the problem. I''ll attach a whole collection of > logs later. > > Thanks, > -- > Radosław Szkodziński
Radoslaw Szkodzinski
2012-Jun-25 15:32 UTC
Re: PCI Passthrough, Radeon 7950 and Windows 7 64-bit
On Mon, Jun 25, 2012 at 5:21 PM, Matthias <matthias.kannenberg@googlemail.com> wrote:> Maybe you should give xm a try just to see if it does the trick. I > never got vga passthrough working with xl (and from my understanding, > it's a lot mor complicated there with compiling the vga bios into xen > and manual calculating vga adress ranges.. with xm, I'm doing neither > of it).Why? I'm not using the VGA Passthrough, The card is set up as a secondary. That shouldn't need any VGA BIOS. Also, it works fine for the first boot very well! The problem occurs on the second start of the same VM.> Also, do you increase the log level for xen? my kernel line is: > multiboot /boot/xen.gz dom0_mem=2048M iommu=1 loglvl=all guest_loglvl=allWill do. (except dom0_mem)> What kernel are you using? If you want i can provide my build commands > for the xen-patched openSuse Kernel..I don't want to use the XenLinux kernel. PVOps only please. Unless the Xen kernel is actually Pvops-based, in which case why would I want to use OpenSUSE one instead of vanilla? As I've mentioned, vanilla 3.4.1. -- Radosław Szkodziński _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
jocelyn falempe
2012-Jun-25 19:51 UTC
Re: PCI Passthrough, Radeon 7950 and Windows 7 64-bit
Hi, I''ve got a similar setup : core i7-3770 + ASRock z77 pro 4M and a sapphire radeon HD7950 Ubuntu 12.04 as Dom0, and Win8 release preview as HVM guest. I can start my VM multiple time without issue. as I have 3 USB controller and 2 SATA controller, I use pci passthrough to pass USB and SATA devices too so my razer mouse, and secondary sata drive is "native" on windows too. here is my script (copied from someone) to reserve the PCI devices for my HVM : remove_device () { BDF=$1 # Unbind a PCI function from its driver as necessary [ ! -e /sys/bus/pci/devices/$BDF/driver/unbind ] || \ echo -n $BDF > /sys/bus/pci/devices/$BDF/driver/unbind # Add a new slot to the PCI Backend''s list echo -n $BDF > /sys/bus/pci/drivers/pciback/new_slot # Now that the backend is watching for the slot, bind to it echo -n $BDF > /sys/bus/pci/drivers/pciback/bind } #USB Controller : remove_device "0000:00:1a.0" #SATA Controller : remove_device "0000:04:00.0" #Radeon 7950 remove_device "0000:01:00.0" #Radeon 7950 audio remove_device "0000:01:00.1" and here is my .cfg file to launch the VM (xm create win8_lan.cfg) : cat win8_lan.cfg kernel="/usr/lib/xen-default/boot/hvmloader" builder=''hvm'' memory = 8096 vcpus=6 name = "win8_new" vif = [ ''type=ioemu, bridge=br0'' ] disk = [''file:/xen-images/win8_2.img,ioemu:hda,w''] acpi = 1 boot="c" sdl=0 serial=''pty'' vnc=1 pci=[ ''00:1a.0'', ''01:00.0'', ''01:00.1'', ''04:00.0'' ] hope it helps Regards, Jocelyn On 06/25/2012 05:32 PM, Radoslaw Szkodzinski wrote:> On Mon, Jun 25, 2012 at 5:21 PM, Matthias > <matthias.kannenberg@googlemail.com> wrote: >> Maybe you should give xm a try just to see if it does the trick. I >> never got vga passthrough working with xl (and from my understanding, >> it''s a lot mor complicated there with compiling the vga bios into xen >> and manual calculating vga adress ranges.. with xm, I''m doing neither >> of it). > Why? I''m not using the VGA Passthrough, The card is set up as a secondary. > That shouldn''t need any VGA BIOS. Also, it works fine for the first > boot very well! > > The problem occurs on the second start of the same VM. > >> Also, do you increase the log level for xen? my kernel line is: >> multiboot /boot/xen.gz dom0_mem=2048M iommu=1 loglvl=all guest_loglvl=all > Will do. (except dom0_mem) > >> What kernel are you using? If you want i can provide my build commands >> for the xen-patched openSuse Kernel.. > I don''t want to use the XenLinux kernel. PVOps only please. > Unless the Xen kernel is actually Pvops-based, in which case why would > I want to use OpenSUSE one instead of vanilla? > > As I''ve mentioned, vanilla 3.4.1. >
Hello, I am fairly certain you are experiencing the second-boot degradation, probably combined with a half-working driver installation. I will try to explain it but the topic is fairly complex and difficult to comprehend without a solid understanding of what is happening. I have attached a flowchart as a visual aid to depict the problem. When Xen, the physical machine, boots the card is passed and the state remains "uninitialized". If the card were to be initialized normally, rebooting the physical machine would reset the state due to a change in power supplied to the card. So the question to ask is how does a virtual machine reset physical hardware? Tobias Geiger wrote up an email on the subject of FLR (Function Level Reset), which I believe is the virtual machine solution to resetting device state. Windows fails to issue an FLR to passed GPU''s. The first pass to Windows works great because it is the first time the card is "initialized", but Windows does not reset the device when you shutdown or restart it. With me so far? His email chains also suggested a solution, which was to use the safely remove hardware menu when the system has started back up, this restores the state of the card (your monitor will go black for a few seconds as the card resets). However, now you are stuck at the crossroads because you can''t get to the second restart due to a BlueScreen. I would be almost certain that you are receiving a bluescreen because the driver installation was run on a second boot previously, where the card state had been initialized during a previous boot of the system. This "first boot" is still under investigation, I believe it initializes if you pass it to another VM, or if you pass it during the installation process, but I haven''t spent the time to verify this. Most people will boot into safe mode and remove the drivers via VNC Console, but if you do this during a second boot without resetting the card first, you will end up with leftover .NET data that prevents ATI Driver Installation AND cannot be removed, meaning you would need to reinstall Windows. So, my suggestion to you is to reboot the physical system, boot Windows and remove the drivers. Reboot the physical system a second time to be sure, and install the drivers again during first boot of Windows. Then give it another spin. If this did not make sense, please let me know where I lost you and I will try to explain it better. ~Casey On Mon, Jun 25, 2012 at 11:32 AM, Radoslaw Szkodzinski < astralstorm@gmail.com> wrote:> On Mon, Jun 25, 2012 at 5:21 PM, Matthias > <matthias.kannenberg@googlemail.com> wrote: > > Maybe you should give xm a try just to see if it does the trick. I > > never got vga passthrough working with xl (and from my understanding, > > it''s a lot mor complicated there with compiling the vga bios into xen > > and manual calculating vga adress ranges.. with xm, I''m doing neither > > of it). > > Why? I''m not using the VGA Passthrough, The card is set up as a secondary. > That shouldn''t need any VGA BIOS. Also, it works fine for the first > boot very well! > > The problem occurs on the second start of the same VM. > > > Also, do you increase the log level for xen? my kernel line is: > > multiboot /boot/xen.gz dom0_mem=2048M iommu=1 loglvl=all guest_loglvl=all > > Will do. (except dom0_mem) > > > What kernel are you using? If you want i can provide my build commands > > for the xen-patched openSuse Kernel.. > > I don''t want to use the XenLinux kernel. PVOps only please. > Unless the Xen kernel is actually Pvops-based, in which case why would > I want to use OpenSUSE one instead of vanilla? > > As I''ve mentioned, vanilla 3.4.1. > > -- > Radosław Szkodziński > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Interesting explanation, but wouldn't that imply that the domU is the only thing responsible for resetting the graphic card? I'm asking because I can simply kill my domU via xm destroy and still be able to boot the domU just fine afterwards (actually, this was an issue up till 3 or so month ago but then was fixed in one of the change sets in the testing repo). i actually remember an IRQ warning after killing the domU,but xen can recover this, so if i understand your explanation correctly, there have to be a mechanism within the dom0 to reset pci devices, too.. 2012/6/26 Casey DeLorme <cdelorme@gmail.com>:> Hello, > > I am fairly certain you are experiencing the second-boot degradation, > probably combined with a half-working driver installation. > > I will try to explain it but the topic is fairly complex and difficult to > comprehend without a solid understanding of what is happening. > > I have attached a flowchart as a visual aid to depict the problem. > > > When Xen, the physical machine, boots the card is passed and the state > remains "uninitialized". If the card were to be initialized normally, > rebooting the physical machine would reset the state due to a change in > power supplied to the card. > > So the question to ask is how does a virtual machine reset physical > hardware? Tobias Geiger wrote up an email on the subject of FLR (Function > Level Reset), which I believe is the virtual machine solution to resetting > device state. > > Windows fails to issue an FLR to passed GPU's. > > > The first pass to Windows works great because it is the first time the card > is "initialized", but Windows does not reset the device when you shutdown or > restart it. > > With me so far? > > His email chains also suggested a solution, which was to use the safely > remove hardware menu when the system has started back up, this restores the > state of the card (your monitor will go black for a few seconds as the card > resets). > > > However, now you are stuck at the crossroads because you can't get to the > second restart due to a BlueScreen. > > I would be almost certain that you are receiving a bluescreen because the > driver installation was run on a second boot previously, where the card > state had been initialized during a previous boot of the system. > > This "first boot" is still under investigation, I believe it initializes if > you pass it to another VM, or if you pass it during the installation > process, but I haven't spent the time to verify this. > > > Most people will boot into safe mode and remove the drivers via VNC Console, > but if you do this during a second boot without resetting the card first, > you will end up with leftover .NET data that prevents ATI Driver > Installation AND cannot be removed, meaning you would need to reinstall > Windows. > > > > So, my suggestion to you is to reboot the physical system, boot Windows and > remove the drivers. Reboot the physical system a second time to be sure, > and install the drivers again during first boot of Windows. > > Then give it another spin. > > If this did not make sense, please let me know where I lost you and I will > try to explain it better. > > ~Casey > > On Mon, Jun 25, 2012 at 11:32 AM, Radoslaw Szkodzinski > <astralstorm@gmail.com> wrote: >> >> On Mon, Jun 25, 2012 at 5:21 PM, Matthias >> <matthias.kannenberg@googlemail.com> wrote: >> > Maybe you should give xm a try just to see if it does the trick. I >> > never got vga passthrough working with xl (and from my understanding, >> > it's a lot mor complicated there with compiling the vga bios into xen >> > and manual calculating vga adress ranges.. with xm, I'm doing neither >> > of it). >> >> Why? I'm not using the VGA Passthrough, The card is set up as a secondary. >> That shouldn't need any VGA BIOS. Also, it works fine for the first >> boot very well! >> >> The problem occurs on the second start of the same VM. >> >> > Also, do you increase the log level for xen? my kernel line is: >> > multiboot /boot/xen.gz dom0_mem=2048M iommu=1 loglvl=all >> > guest_loglvl=all >> >> Will do. (except dom0_mem) >> >> > What kernel are you using? If you want i can provide my build commands >> > for the xen-patched openSuse Kernel.. >> >> I don't want to use the XenLinux kernel. PVOps only please. >> Unless the Xen kernel is actually Pvops-based, in which case why would >> I want to use OpenSUSE one instead of vanilla? >> >> As I've mentioned, vanilla 3.4.1. >> >> -- >> Radosław Szkodziński >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xen.org >> http://lists.xen.org/xen-users > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
If you are passing the card to xen-pciback then Dom0 has nothing to do with the device. Short of physically resetting the device via power-cycling the physical system, then yes that is exactly what I am implying. Please understand, I only ran tests with Windows, I have yet to setup passthrough with Linux or Unix, so the results may vary if you are using aa DomU with a different operating system. To my understanding Xen does not inform the card that it needs to reset, I am pretty sure it was remarked in the aforementioned email chain that this is the responsibility of the DomU. I can only hazard a guess as to why, either it isn''t sending one because Windows isn''t exactly advocating running their system as a Virtual Machine, OR because we are using secondary passthrough the card is not available at early boot time, and perhaps that is when Windows is issuing a reset. Again, no facts supporting either of those, only wild speculation. I don''t think IRQ is relative but I could be wrong. To my understanding IRQ gives the OS a way to send signals to the card, it doesn''t tell the OS what signals to send to the card, which would mean the OS chooses whether to send the reset signal. If the card is passed using xen-pciback then Dom0 never bothers the device. With late binding this may be an entirely different story. I have yet to hear of a Windows Dom0, so again not something my tests can yield conclusions on. However, to my understanding Dom0 is tied to Xen, I haven''t heard of anyone restarting Dom0 without rebooting the physical machine, and going back to our original logic, resetting the machine changes the power which physically resets devices attached to the machine (this would include devices sent to xen-pciback). Another speculation of mine is that the reason behind the performance drop is at second initialization the card has been told to serve two owners, which throws a wrench into its operations. I am no expert, I am only supplying my attempt at drawing a logical conclusion from the problem and my tests. I could supply the exact same flowchart without terms like FLR and device State, and it wouldn''t change as the process and logic remain the same. Let me know if that helps clear things up any. On Mon, Jun 25, 2012 at 7:29 PM, Matthias < matthias.kannenberg@googlemail.com> wrote:> Interesting explanation, but wouldn''t that imply that the domU is the > only thing responsible for resetting the graphic card? > I''m asking because I can simply kill my domU via xm destroy and still > be able to boot the domU just fine afterwards (actually, this was an > issue up till 3 or so month ago but then was fixed in one of the > change sets in the testing repo). > > i actually remember an IRQ warning after killing the domU,but xen can > recover this, so if i understand your explanation correctly, there > have to be a mechanism within the dom0 to reset pci devices, too.. > > 2012/6/26 Casey DeLorme <cdelorme@gmail.com>: > > Hello, > > > > I am fairly certain you are experiencing the second-boot degradation, > > probably combined with a half-working driver installation. > > > > I will try to explain it but the topic is fairly complex and difficult to > > comprehend without a solid understanding of what is happening. > > > > I have attached a flowchart as a visual aid to depict the problem. > > > > > > When Xen, the physical machine, boots the card is passed and the state > > remains "uninitialized". If the card were to be initialized normally, > > rebooting the physical machine would reset the state due to a change in > > power supplied to the card. > > > > So the question to ask is how does a virtual machine reset physical > > hardware? Tobias Geiger wrote up an email on the subject of FLR > (Function > > Level Reset), which I believe is the virtual machine solution to > resetting > > device state. > > > > Windows fails to issue an FLR to passed GPU''s. > > > > > > The first pass to Windows works great because it is the first time the > card > > is "initialized", but Windows does not reset the device when you > shutdown or > > restart it. > > > > With me so far? > > > > His email chains also suggested a solution, which was to use the safely > > remove hardware menu when the system has started back up, this restores > the > > state of the card (your monitor will go black for a few seconds as the > card > > resets). > > > > > > However, now you are stuck at the crossroads because you can''t get to the > > second restart due to a BlueScreen. > > > > I would be almost certain that you are receiving a bluescreen because the > > driver installation was run on a second boot previously, where the card > > state had been initialized during a previous boot of the system. > > > > This "first boot" is still under investigation, I believe it initializes > if > > you pass it to another VM, or if you pass it during the installation > > process, but I haven''t spent the time to verify this. > > > > > > Most people will boot into safe mode and remove the drivers via VNC > Console, > > but if you do this during a second boot without resetting the card first, > > you will end up with leftover .NET data that prevents ATI Driver > > Installation AND cannot be removed, meaning you would need to reinstall > > Windows. > > > > > > > > So, my suggestion to you is to reboot the physical system, boot Windows > and > > remove the drivers. Reboot the physical system a second time to be sure, > > and install the drivers again during first boot of Windows. > > > > Then give it another spin. > > > > If this did not make sense, please let me know where I lost you and I > will > > try to explain it better. > > > > ~Casey > > > > On Mon, Jun 25, 2012 at 11:32 AM, Radoslaw Szkodzinski > > <astralstorm@gmail.com> wrote: > >> > >> On Mon, Jun 25, 2012 at 5:21 PM, Matthias > >> <matthias.kannenberg@googlemail.com> wrote: > >> > Maybe you should give xm a try just to see if it does the trick. I > >> > never got vga passthrough working with xl (and from my understanding, > >> > it''s a lot mor complicated there with compiling the vga bios into xen > >> > and manual calculating vga adress ranges.. with xm, I''m doing neither > >> > of it). > >> > >> Why? I''m not using the VGA Passthrough, The card is set up as a > secondary. > >> That shouldn''t need any VGA BIOS. Also, it works fine for the first > >> boot very well! > >> > >> The problem occurs on the second start of the same VM. > >> > >> > Also, do you increase the log level for xen? my kernel line is: > >> > multiboot /boot/xen.gz dom0_mem=2048M iommu=1 loglvl=all > >> > guest_loglvl=all > >> > >> Will do. (except dom0_mem) > >> > >> > What kernel are you using? If you want i can provide my build commands > >> > for the xen-patched openSuse Kernel.. > >> > >> I don''t want to use the XenLinux kernel. PVOps only please. > >> Unless the Xen kernel is actually Pvops-based, in which case why would > >> I want to use OpenSUSE one instead of vanilla? > >> > >> As I''ve mentioned, vanilla 3.4.1. > >> > >> -- > >> Radosław Szkodziński > >> > >> _______________________________________________ > >> Xen-users mailing list > >> Xen-users@lists.xen.org > >> http://lists.xen.org/xen-users > > > > > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xen.org > > http://lists.xen.org/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Hi, sorry if i was unclear: I will test this tomorow, but i''m pretty sure i can kill my windows 7 domU (has vga, usb controller and onboard sound via pciback) via xm and afterwards booting it just fine again with everything working like normal (also possible with linux domUs).. It might be right that the dom0 can''t actually use the hidden pci devices, but I think that at least the xen itself has some abilities to control it (hence for example listing the assignable devices via xm in my opinion indicates an awareness). But again, these are just my two cents on your theory.. i will test this tomorrow with my setup.. 2012/6/26 Casey DeLorme <cdelorme@gmail.com>:> > If you are passing the card to xen-pciback then Dom0 has nothing to do with > the device. Short of physically resetting the device via power-cycling the > physical system, then yes that is exactly what I am implying. > > Please understand, I only ran tests with Windows, I have yet to setup > passthrough with Linux or Unix, so the results may vary if you are using aa > DomU with a different operating system. > > > To my understanding Xen does not inform the card that it needs to reset, I > am pretty sure it was remarked in the aforementioned email chain that this > is the responsibility of the DomU. I can only hazard a guess as to why, > either it isn''t sending one because Windows isn''t exactly advocating running > their system as a Virtual Machine, OR because we are using secondary > passthrough the card is not available at early boot time, and perhaps that > is when Windows is issuing a reset. Again, no facts supporting either of > those, only wild speculation. > > > I don''t think IRQ is relative but I could be wrong. To my understanding IRQ > gives the OS a way to send signals to the card, it doesn''t tell the OS what > signals to send to the card, which would mean the OS chooses whether to send > the reset signal. > > If the card is passed using xen-pciback then Dom0 never bothers the device. > With late binding this may be an entirely different story. I have yet to > hear of a Windows Dom0, so again not something my tests can yield > conclusions on. However, to my understanding Dom0 is tied to Xen, I haven''t > heard of anyone restarting Dom0 without rebooting the physical machine, and > going back to our original logic, resetting the machine changes the power > which physically resets devices attached to the machine (this would include > devices sent to xen-pciback). > > > Another speculation of mine is that the reason behind the performance drop > is at second initialization the card has been told to serve two owners, > which throws a wrench into its operations. > > > I am no expert, I am only supplying my attempt at drawing a logical > conclusion from the problem and my tests. I could supply the exact same > flowchart without terms like FLR and device State, and it wouldn''t change as > the process and logic remain the same. > > Let me know if that helps clear things up any. > > > On Mon, Jun 25, 2012 at 7:29 PM, Matthias > <matthias.kannenberg@googlemail.com> wrote: >> >> Interesting explanation, but wouldn''t that imply that the domU is the >> only thing responsible for resetting the graphic card? >> I''m asking because I can simply kill my domU via xm destroy and still >> be able to boot the domU just fine afterwards (actually, this was an >> issue up till 3 or so month ago but then was fixed in one of the >> change sets in the testing repo). >> >> i actually remember an IRQ warning after killing the domU,but xen can >> recover this, so if i understand your explanation correctly, there >> have to be a mechanism within the dom0 to reset pci devices, too.. >> >> 2012/6/26 Casey DeLorme <cdelorme@gmail.com>: >> > Hello, >> > >> > I am fairly certain you are experiencing the second-boot degradation, >> > probably combined with a half-working driver installation. >> > >> > I will try to explain it but the topic is fairly complex and difficult >> > to >> > comprehend without a solid understanding of what is happening. >> > >> > I have attached a flowchart as a visual aid to depict the problem. >> > >> > >> > When Xen, the physical machine, boots the card is passed and the state >> > remains "uninitialized". If the card were to be initialized normally, >> > rebooting the physical machine would reset the state due to a change in >> > power supplied to the card. >> > >> > So the question to ask is how does a virtual machine reset physical >> > hardware? Tobias Geiger wrote up an email on the subject of FLR >> > (Function >> > Level Reset), which I believe is the virtual machine solution to >> > resetting >> > device state. >> > >> > Windows fails to issue an FLR to passed GPU''s. >> > >> > >> > The first pass to Windows works great because it is the first time the >> > card >> > is "initialized", but Windows does not reset the device when you >> > shutdown or >> > restart it. >> > >> > With me so far? >> > >> > His email chains also suggested a solution, which was to use the safely >> > remove hardware menu when the system has started back up, this restores >> > the >> > state of the card (your monitor will go black for a few seconds as the >> > card >> > resets). >> > >> > >> > However, now you are stuck at the crossroads because you can''t get to >> > the >> > second restart due to a BlueScreen. >> > >> > I would be almost certain that you are receiving a bluescreen because >> > the >> > driver installation was run on a second boot previously, where the card >> > state had been initialized during a previous boot of the system. >> > >> > This "first boot" is still under investigation, I believe it initializes >> > if >> > you pass it to another VM, or if you pass it during the installation >> > process, but I haven''t spent the time to verify this. >> > >> > >> > Most people will boot into safe mode and remove the drivers via VNC >> > Console, >> > but if you do this during a second boot without resetting the card >> > first, >> > you will end up with leftover .NET data that prevents ATI Driver >> > Installation AND cannot be removed, meaning you would need to reinstall >> > Windows. >> > >> > >> > >> > So, my suggestion to you is to reboot the physical system, boot Windows >> > and >> > remove the drivers. Reboot the physical system a second time to be >> > sure, >> > and install the drivers again during first boot of Windows. >> > >> > Then give it another spin. >> > >> > If this did not make sense, please let me know where I lost you and I >> > will >> > try to explain it better. >> > >> > ~Casey >> > >> > On Mon, Jun 25, 2012 at 11:32 AM, Radoslaw Szkodzinski >> > <astralstorm@gmail.com> wrote: >> >> >> >> On Mon, Jun 25, 2012 at 5:21 PM, Matthias >> >> <matthias.kannenberg@googlemail.com> wrote: >> >> > Maybe you should give xm a try just to see if it does the trick. I >> >> > never got vga passthrough working with xl (and from my understanding, >> >> > it''s a lot mor complicated there with compiling the vga bios into xen >> >> > and manual calculating vga adress ranges.. with xm, I''m doing neither >> >> > of it). >> >> >> >> Why? I''m not using the VGA Passthrough, The card is set up as a >> >> secondary. >> >> That shouldn''t need any VGA BIOS. Also, it works fine for the first >> >> boot very well! >> >> >> >> The problem occurs on the second start of the same VM. >> >> >> >> > Also, do you increase the log level for xen? my kernel line is: >> >> > multiboot /boot/xen.gz dom0_mem=2048M iommu=1 loglvl=all >> >> > guest_loglvl=all >> >> >> >> Will do. (except dom0_mem) >> >> >> >> > What kernel are you using? If you want i can provide my build >> >> > commands >> >> > for the xen-patched openSuse Kernel.. >> >> >> >> I don''t want to use the XenLinux kernel. PVOps only please. >> >> Unless the Xen kernel is actually Pvops-based, in which case why would >> >> I want to use OpenSUSE one instead of vanilla? >> >> >> >> As I''ve mentioned, vanilla 3.4.1. >> >> >> >> -- >> >> Radosław Szkodziński >> >> >> >> _______________________________________________ >> >> Xen-users mailing list >> >> Xen-users@lists.xen.org >> >> http://lists.xen.org/xen-users >> > >> > >> > >> > _______________________________________________ >> > Xen-users mailing list >> > Xen-users@lists.xen.org >> > http://lists.xen.org/xen-users > >
I agree with you that Xen has an awareness, but what I read suggested that the DomU is supposed to be responsible for the reset. In any event, please do post your results. If you don''t have the same performance degradation and you can help identify where our configurations differ, it could help fix the problem which would be awesome. I forgot to mention, my Windows can reboot, and the GUI works fine, it''s when I try to run a 3D application like a modern video game that I run into problems. Results vary, The Last Remnant by Square Enix runs extremely slow and usually crashes after a certain point, Borderlands drops from 60~ FPS at max settings to under 10~ FPS and a lower max resolution. On Mon, Jun 25, 2012 at 8:21 PM, Matthias < matthias.kannenberg@googlemail.com> wrote:> Hi, > > sorry if i was unclear: I will test this tomorow, but i''m pretty sure > i can kill my windows 7 domU (has vga, usb controller and onboard > sound via pciback) via xm and afterwards booting it just fine again > with everything working like normal (also possible with linux domUs).. > > It might be right that the dom0 can''t actually use the hidden pci > devices, but I think that at least the xen itself has some abilities > to control it (hence for example listing the assignable devices via xm > in my opinion indicates an awareness). > > But again, these are just my two cents on your theory.. i will test > this tomorrow with my setup.. > > > 2012/6/26 Casey DeLorme <cdelorme@gmail.com>: > > > > If you are passing the card to xen-pciback then Dom0 has nothing to do > with > > the device. Short of physically resetting the device via power-cycling > the > > physical system, then yes that is exactly what I am implying. > > > > Please understand, I only ran tests with Windows, I have yet to setup > > passthrough with Linux or Unix, so the results may vary if you are using > aa > > DomU with a different operating system. > > > > > > To my understanding Xen does not inform the card that it needs to reset, > I > > am pretty sure it was remarked in the aforementioned email chain that > this > > is the responsibility of the DomU. I can only hazard a guess as to why, > > either it isn''t sending one because Windows isn''t exactly advocating > running > > their system as a Virtual Machine, OR because we are using secondary > > passthrough the card is not available at early boot time, and perhaps > that > > is when Windows is issuing a reset. Again, no facts supporting either of > > those, only wild speculation. > > > > > > I don''t think IRQ is relative but I could be wrong. To my understanding > IRQ > > gives the OS a way to send signals to the card, it doesn''t tell the OS > what > > signals to send to the card, which would mean the OS chooses whether to > send > > the reset signal. > > > > If the card is passed using xen-pciback then Dom0 never bothers the > device. > > With late binding this may be an entirely different story. I have yet > to > > hear of a Windows Dom0, so again not something my tests can yield > > conclusions on. However, to my understanding Dom0 is tied to Xen, I > haven''t > > heard of anyone restarting Dom0 without rebooting the physical machine, > and > > going back to our original logic, resetting the machine changes the power > > which physically resets devices attached to the machine (this would > include > > devices sent to xen-pciback). > > > > > > Another speculation of mine is that the reason behind the performance > drop > > is at second initialization the card has been told to serve two owners, > > which throws a wrench into its operations. > > > > > > I am no expert, I am only supplying my attempt at drawing a logical > > conclusion from the problem and my tests. I could supply the exact same > > flowchart without terms like FLR and device State, and it wouldn''t > change as > > the process and logic remain the same. > > > > Let me know if that helps clear things up any. > > > > > > On Mon, Jun 25, 2012 at 7:29 PM, Matthias > > <matthias.kannenberg@googlemail.com> wrote: > >> > >> Interesting explanation, but wouldn''t that imply that the domU is the > >> only thing responsible for resetting the graphic card? > >> I''m asking because I can simply kill my domU via xm destroy and still > >> be able to boot the domU just fine afterwards (actually, this was an > >> issue up till 3 or so month ago but then was fixed in one of the > >> change sets in the testing repo). > >> > >> i actually remember an IRQ warning after killing the domU,but xen can > >> recover this, so if i understand your explanation correctly, there > >> have to be a mechanism within the dom0 to reset pci devices, too.. > >> > >> 2012/6/26 Casey DeLorme <cdelorme@gmail.com>: > >> > Hello, > >> > > >> > I am fairly certain you are experiencing the second-boot degradation, > >> > probably combined with a half-working driver installation. > >> > > >> > I will try to explain it but the topic is fairly complex and difficult > >> > to > >> > comprehend without a solid understanding of what is happening. > >> > > >> > I have attached a flowchart as a visual aid to depict the problem. > >> > > >> > > >> > When Xen, the physical machine, boots the card is passed and the state > >> > remains "uninitialized". If the card were to be initialized normally, > >> > rebooting the physical machine would reset the state due to a change > in > >> > power supplied to the card. > >> > > >> > So the question to ask is how does a virtual machine reset physical > >> > hardware? Tobias Geiger wrote up an email on the subject of FLR > >> > (Function > >> > Level Reset), which I believe is the virtual machine solution to > >> > resetting > >> > device state. > >> > > >> > Windows fails to issue an FLR to passed GPU''s. > >> > > >> > > >> > The first pass to Windows works great because it is the first time the > >> > card > >> > is "initialized", but Windows does not reset the device when you > >> > shutdown or > >> > restart it. > >> > > >> > With me so far? > >> > > >> > His email chains also suggested a solution, which was to use the > safely > >> > remove hardware menu when the system has started back up, this > restores > >> > the > >> > state of the card (your monitor will go black for a few seconds as the > >> > card > >> > resets). > >> > > >> > > >> > However, now you are stuck at the crossroads because you can''t get to > >> > the > >> > second restart due to a BlueScreen. > >> > > >> > I would be almost certain that you are receiving a bluescreen because > >> > the > >> > driver installation was run on a second boot previously, where the > card > >> > state had been initialized during a previous boot of the system. > >> > > >> > This "first boot" is still under investigation, I believe it > initializes > >> > if > >> > you pass it to another VM, or if you pass it during the installation > >> > process, but I haven''t spent the time to verify this. > >> > > >> > > >> > Most people will boot into safe mode and remove the drivers via VNC > >> > Console, > >> > but if you do this during a second boot without resetting the card > >> > first, > >> > you will end up with leftover .NET data that prevents ATI Driver > >> > Installation AND cannot be removed, meaning you would need to > reinstall > >> > Windows. > >> > > >> > > >> > > >> > So, my suggestion to you is to reboot the physical system, boot > Windows > >> > and > >> > remove the drivers. Reboot the physical system a second time to be > >> > sure, > >> > and install the drivers again during first boot of Windows. > >> > > >> > Then give it another spin. > >> > > >> > If this did not make sense, please let me know where I lost you and I > >> > will > >> > try to explain it better. > >> > > >> > ~Casey > >> > > >> > On Mon, Jun 25, 2012 at 11:32 AM, Radoslaw Szkodzinski > >> > <astralstorm@gmail.com> wrote: > >> >> > >> >> On Mon, Jun 25, 2012 at 5:21 PM, Matthias > >> >> <matthias.kannenberg@googlemail.com> wrote: > >> >> > Maybe you should give xm a try just to see if it does the trick. I > >> >> > never got vga passthrough working with xl (and from my > understanding, > >> >> > it''s a lot mor complicated there with compiling the vga bios into > xen > >> >> > and manual calculating vga adress ranges.. with xm, I''m doing > neither > >> >> > of it). > >> >> > >> >> Why? I''m not using the VGA Passthrough, The card is set up as a > >> >> secondary. > >> >> That shouldn''t need any VGA BIOS. Also, it works fine for the first > >> >> boot very well! > >> >> > >> >> The problem occurs on the second start of the same VM. > >> >> > >> >> > Also, do you increase the log level for xen? my kernel line is: > >> >> > multiboot /boot/xen.gz dom0_mem=2048M iommu=1 loglvl=all > >> >> > guest_loglvl=all > >> >> > >> >> Will do. (except dom0_mem) > >> >> > >> >> > What kernel are you using? If you want i can provide my build > >> >> > commands > >> >> > for the xen-patched openSuse Kernel.. > >> >> > >> >> I don''t want to use the XenLinux kernel. PVOps only please. > >> >> Unless the Xen kernel is actually Pvops-based, in which case why > would > >> >> I want to use OpenSUSE one instead of vanilla? > >> >> > >> >> As I''ve mentioned, vanilla 3.4.1. > >> >> > >> >> -- > >> >> Radosław Szkodziński > >> >> > >> >> _______________________________________________ > >> >> Xen-users mailing list > >> >> Xen-users@lists.xen.org > >> >> http://lists.xen.org/xen-users > >> > > >> > > >> > > >> > _______________________________________________ > >> > Xen-users mailing list > >> > Xen-users@lists.xen.org > >> > http://lists.xen.org/xen-users > > > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Radoslaw Szkodzinski
2012-Jun-26 01:13 UTC
Re: PCI Passthrough, Radeon 7950 and Windows 7 64-bit
On Tue, Jun 26, 2012 at 2:51 AM, Casey DeLorme <cdelorme@gmail.com> wrote:> I agree with you that Xen has an awareness, but what I read suggested that > the DomU is supposed to be responsible for the reset.Quite silly, many OSes and drivers don't care about device shutdown on "poweroff". Why should they, the power usually will be off real soon anyway. I think currently only Linux cares enough - due to the need to support kexec. Suspend to ram is different matter. Perhaps that avenue would be useful to explore. (Attempting to S2R the VM instead of shutdown...) Still, it'd be nice for Xen to force reset the devices (FLR, then D0->D3->D0 ASPM, then finally PCI bus reset as a last resort) when the VM stops using the devices. Better safe than sorry.> In any event, please > do post your results. If you don't have the same performance degradation > and you can help identify where our configurations differ, it could help fix > the problem which would be awesome.-- Radosław Szkodziński _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Hi, did some testing in my lunch break: # dmesg -c && xm dmesg -c # xm destroy WINDOMU # dmesg [257342.161103] xen1: port 1(work) entered disabled state [257342.162777] xen1: port 1(work) entered disabled state [257345.003372] irq 18: nobody cared (try booting with the "irqpoll" option) [257345.004847] Pid: 11048, comm: qemu-dm Not tainted 3.4.2-xen #2 [257345.006208] Call Trace: [257345.006208] [<ffffffff800085d5>] dump_trace+0x85/0x1c0 [257345.006208] [<ffffffff8088c036>] dump_stack+0x69/0x6f [257345.006208] [<ffffffff800c9936>] __report_bad_irq+0x36/0xe0 [257345.006208] [<ffffffff800c9c8b>] note_interrupt+0x1fb/0x240 [257345.006208] [<ffffffff800c72c4>] handle_irq_event_percpu+0x94/0x1d0 [257345.006208] [<ffffffff800c7464>] handle_irq_event+0x64/0x90 [257345.006208] [<ffffffff800ca7b4>] handle_fasteoi_irq+0x64/0x120 [257345.006208] [<ffffffff80008468>] handle_irq+0x18/0x30 [257345.006208] [<ffffffff805c1cc4>] evtchn_do_upcall+0x1c4/0x2e0 [257345.006208] [<ffffffff808a308e>] do_hypervisor_callback+0x1e/0x30 [257345.006208] [<ffffffff8014802a>] fget_light+0x3a/0xc0 [257345.006208] [<ffffffff80159c65>] do_select+0x315/0x660 [257345.006208] [<ffffffff8015a15a>] core_sys_select+0x1aa/0x2e0 [257345.006208] [<ffffffff8015a347>] sys_select+0xb7/0x110 [257345.006208] [<ffffffff808a29bb>] system_call_fastpath+0x1a/0x1f [257345.006208] [<00007f1a6d0081d3>] 0x7f1a6d0081d2 [257345.006208] handlers: [257345.006208] [<ffffffff80658a80>] usb_hcd_irq [257345.006208] [<ffffffff80658a80>] usb_hcd_irq [257345.006208] [<ffffffff80658a80>] usb_hcd_irq [257345.006208] Disabling IRQ #18 # xm dmesg <nothing here> # xm create WINDOMU booted without problems and i played some Diablo 3 for testing purpose: no performance change, everything normal Then i analysed the kernel output in dmesg from the kill and found out that the IRQ 18 is not in fact my graphic card what i always thought, but my usb host controler which i have forwarded to the domU, too. So appereantly there is no nothing vga related in the dmesg. Then I thought okay, if the dom0 in fact doesn''t care about the vga state, it might actually be the domU and the only thing i guess can be responsible for that is this in my domU config: ####################### # PCI Power Management: # # If it''s set, the guest OS will be able to program D0-D3hot states of the # PCI device for the purpose of low power consumption. # pci_power_mgmt=1 ####################### My new theory is that this allowes the domU (and therefor my windows) to reset the vga on boot so i don''t have the problems you have.. Question is: Are you using this option in your domU or might that be the difference in our configs? 2012/6/26 Radoslaw Szkodzinski <astralstorm@gmail.com>:> On Tue, Jun 26, 2012 at 2:51 AM, Casey DeLorme <cdelorme@gmail.com> wrote: >> I agree with you that Xen has an awareness, but what I read suggested that >> the DomU is supposed to be responsible for the reset. > > Quite silly, many OSes and drivers don''t care about device shutdown on > "poweroff". > Why should they, the power usually will be off real soon anyway. > I think currently only Linux cares enough - due to the need to support kexec. > > Suspend to ram is different matter. Perhaps that avenue would be > useful to explore. > (Attempting to S2R the VM instead of shutdown...) > > Still, it''d be nice for Xen to force reset the devices (FLR, then > D0->D3->D0 ASPM, then finally PCI bus reset as a last resort) when the > VM stops using the devices. > Better safe than sorry. > >> In any event, please >> do post your results. If you don''t have the same performance degradation >> and you can help identify where our configurations differ, it could help fix >> the problem which would be awesome. > > -- > Radosław Szkodziński
Radoslaw Szkodzinski
2012-Jun-26 14:37 UTC
Re: PCI Passthrough, Radeon 7950 and Windows 7 64-bit
Xm seems to work fine. I remember it has its own PCI reset code when destroying the VM, which calls the FLR first. Or none at all. Xl on the other hand forces a PCI bus reset, causing the hard hang. The PAGE_FAULT only happens actually with the older kernel. It seems that xl uses dom0 kernel to do a PCI reset. pci_power_mgmt option does nothing particular. However, I think my VM image got damaged from multiple hard crashes and I''ll have to reinstall. I''m getting really weird issues with it now, such as bluescreens in xenvbd.sys. Will check it again after a reinstall.
I am using the xl toolstack, with Xen 4.2. That''s probably the difference, I tried the pci flags (both inline and stand-alone versions of them) without any changes. On Tue, Jun 26, 2012 at 7:19 AM, Matthias < matthias.kannenberg@googlemail.com> wrote:> Hi, > > did some testing in my lunch break: > > # dmesg -c && xm dmesg -c > # xm destroy WINDOMU > # dmesg > [257342.161103] xen1: port 1(work) entered disabled state > [257342.162777] xen1: port 1(work) entered disabled state > [257345.003372] irq 18: nobody cared (try booting with the "irqpoll" > option) > [257345.004847] Pid: 11048, comm: qemu-dm Not tainted 3.4.2-xen #2 > [257345.006208] Call Trace: > [257345.006208] [<ffffffff800085d5>] dump_trace+0x85/0x1c0 > [257345.006208] [<ffffffff8088c036>] dump_stack+0x69/0x6f > [257345.006208] [<ffffffff800c9936>] __report_bad_irq+0x36/0xe0 > [257345.006208] [<ffffffff800c9c8b>] note_interrupt+0x1fb/0x240 > [257345.006208] [<ffffffff800c72c4>] handle_irq_event_percpu+0x94/0x1d0 > [257345.006208] [<ffffffff800c7464>] handle_irq_event+0x64/0x90 > [257345.006208] [<ffffffff800ca7b4>] handle_fasteoi_irq+0x64/0x120 > [257345.006208] [<ffffffff80008468>] handle_irq+0x18/0x30 > [257345.006208] [<ffffffff805c1cc4>] evtchn_do_upcall+0x1c4/0x2e0 > [257345.006208] [<ffffffff808a308e>] do_hypervisor_callback+0x1e/0x30 > [257345.006208] [<ffffffff8014802a>] fget_light+0x3a/0xc0 > [257345.006208] [<ffffffff80159c65>] do_select+0x315/0x660 > [257345.006208] [<ffffffff8015a15a>] core_sys_select+0x1aa/0x2e0 > [257345.006208] [<ffffffff8015a347>] sys_select+0xb7/0x110 > [257345.006208] [<ffffffff808a29bb>] system_call_fastpath+0x1a/0x1f > [257345.006208] [<00007f1a6d0081d3>] 0x7f1a6d0081d2 > [257345.006208] handlers: > [257345.006208] [<ffffffff80658a80>] usb_hcd_irq > [257345.006208] [<ffffffff80658a80>] usb_hcd_irq > [257345.006208] [<ffffffff80658a80>] usb_hcd_irq > [257345.006208] Disabling IRQ #18 > # xm dmesg > <nothing here> > # xm create WINDOMU > booted without problems and i played some Diablo 3 for testing > purpose: no performance change, everything normal > > Then i analysed the kernel output in dmesg from the kill and found out > that the IRQ 18 is not in fact my graphic card what i always thought, > but my usb host controler which i have forwarded to the domU, too. So > appereantly there is no nothing vga related in the dmesg. Then I > thought okay, if the dom0 in fact doesn''t care about the vga state, it > might actually be the domU and the only thing i guess can be > responsible for that is this in my domU config: > > ####################### > # PCI Power Management: > # > # If it''s set, the guest OS will be able to program D0-D3hot states of > the > # PCI device for the purpose of low power consumption. > # > pci_power_mgmt=1 > ####################### > > My new theory is that this allowes the domU (and therefor my windows) > to reset the vga on boot so i don''t have the problems you have.. > > Question is: Are you using this option in your domU or might that be > the difference in our configs? > > > > > 2012/6/26 Radoslaw Szkodzinski <astralstorm@gmail.com>: > > On Tue, Jun 26, 2012 at 2:51 AM, Casey DeLorme <cdelorme@gmail.com> > wrote: > >> I agree with you that Xen has an awareness, but what I read suggested > that > >> the DomU is supposed to be responsible for the reset. > > > > Quite silly, many OSes and drivers don''t care about device shutdown on > > "poweroff". > > Why should they, the power usually will be off real soon anyway. > > I think currently only Linux cares enough - due to the need to support > kexec. > > > > Suspend to ram is different matter. Perhaps that avenue would be > > useful to explore. > > (Attempting to S2R the VM instead of shutdown...) > > > > Still, it''d be nice for Xen to force reset the devices (FLR, then > > D0->D3->D0 ASPM, then finally PCI bus reset as a last resort) when the > > VM stops using the devices. > > Better safe than sorry. > > > >> In any event, please > >> do post your results. If you don''t have the same performance > degradation > >> and you can help identify where our configurations differ, it could > help fix > >> the problem which would be awesome. > > > > -- > > Radosław Szkodziński >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Well, then as i said before: I thing xm is a lot more stable right now then xl and i don't really see any benefit of using xl right now. Actually I tried using xl instead of xm last week and but run intu issues with creating vifs when using nat. Interesting to see that there are also issues with vga passthrough. Are there - right now - any benefits of using xl instead of xm? I thought that it's easier to run stubdoms with xl, but after reading that stubdoms don't bring you any significant performance increate when using PVOPS-HVMs, i gave up on trieing to get stubdoms running. So it would be nice to here if migrating to xl is actually something to look into in the (near) future. 2012/6/27 Casey DeLorme <cdelorme@gmail.com>:> I am using the xl toolstack, with Xen 4.2. That's probably the difference, > I tried the pci flags (both inline and stand-alone versions of them) without > any changes. > > On Tue, Jun 26, 2012 at 7:19 AM, Matthias > <matthias.kannenberg@googlemail.com> wrote: >> >> Hi, >> >> did some testing in my lunch break: >> >> # dmesg -c && xm dmesg -c >> # xm destroy WINDOMU >> # dmesg >> [257342.161103] xen1: port 1(work) entered disabled state >> [257342.162777] xen1: port 1(work) entered disabled state >> [257345.003372] irq 18: nobody cared (try booting with the "irqpoll" >> option) >> [257345.004847] Pid: 11048, comm: qemu-dm Not tainted 3.4.2-xen #2 >> [257345.006208] Call Trace: >> [257345.006208] [<ffffffff800085d5>] dump_trace+0x85/0x1c0 >> [257345.006208] [<ffffffff8088c036>] dump_stack+0x69/0x6f >> [257345.006208] [<ffffffff800c9936>] __report_bad_irq+0x36/0xe0 >> [257345.006208] [<ffffffff800c9c8b>] note_interrupt+0x1fb/0x240 >> [257345.006208] [<ffffffff800c72c4>] handle_irq_event_percpu+0x94/0x1d0 >> [257345.006208] [<ffffffff800c7464>] handle_irq_event+0x64/0x90 >> [257345.006208] [<ffffffff800ca7b4>] handle_fasteoi_irq+0x64/0x120 >> [257345.006208] [<ffffffff80008468>] handle_irq+0x18/0x30 >> [257345.006208] [<ffffffff805c1cc4>] evtchn_do_upcall+0x1c4/0x2e0 >> [257345.006208] [<ffffffff808a308e>] do_hypervisor_callback+0x1e/0x30 >> [257345.006208] [<ffffffff8014802a>] fget_light+0x3a/0xc0 >> [257345.006208] [<ffffffff80159c65>] do_select+0x315/0x660 >> [257345.006208] [<ffffffff8015a15a>] core_sys_select+0x1aa/0x2e0 >> [257345.006208] [<ffffffff8015a347>] sys_select+0xb7/0x110 >> [257345.006208] [<ffffffff808a29bb>] system_call_fastpath+0x1a/0x1f >> [257345.006208] [<00007f1a6d0081d3>] 0x7f1a6d0081d2 >> [257345.006208] handlers: >> [257345.006208] [<ffffffff80658a80>] usb_hcd_irq >> [257345.006208] [<ffffffff80658a80>] usb_hcd_irq >> [257345.006208] [<ffffffff80658a80>] usb_hcd_irq >> [257345.006208] Disabling IRQ #18 >> # xm dmesg >> <nothing here> >> # xm create WINDOMU >> booted without problems and i played some Diablo 3 for testing >> purpose: no performance change, everything normal >> >> Then i analysed the kernel output in dmesg from the kill and found out >> that the IRQ 18 is not in fact my graphic card what i always thought, >> but my usb host controler which i have forwarded to the domU, too. So >> appereantly there is no nothing vga related in the dmesg. Then I >> thought okay, if the dom0 in fact doesn't care about the vga state, it >> might actually be the domU and the only thing i guess can be >> responsible for that is this in my domU config: >> >> ####################### >> # PCI Power Management: >> # >> # If it's set, the guest OS will be able to program D0-D3hot states of >> the >> # PCI device for the purpose of low power consumption. >> # >> pci_power_mgmt=1 >> ####################### >> >> My new theory is that this allowes the domU (and therefor my windows) >> to reset the vga on boot so i don't have the problems you have.. >> >> Question is: Are you using this option in your domU or might that be >> the difference in our configs? >> >> >> >> >> 2012/6/26 Radoslaw Szkodzinski <astralstorm@gmail.com>: >> > On Tue, Jun 26, 2012 at 2:51 AM, Casey DeLorme <cdelorme@gmail.com> >> > wrote: >> >> I agree with you that Xen has an awareness, but what I read suggested >> >> that >> >> the DomU is supposed to be responsible for the reset. >> > >> > Quite silly, many OSes and drivers don't care about device shutdown on >> > "poweroff". >> > Why should they, the power usually will be off real soon anyway. >> > I think currently only Linux cares enough - due to the need to support >> > kexec. >> > >> > Suspend to ram is different matter. Perhaps that avenue would be >> > useful to explore. >> > (Attempting to S2R the VM instead of shutdown...) >> > >> > Still, it'd be nice for Xen to force reset the devices (FLR, then >> > D0->D3->D0 ASPM, then finally PCI bus reset as a last resort) when the >> > VM stops using the devices. >> > Better safe than sorry. >> > >> >> In any event, please >> >> do post your results. If you don't have the same performance >> >> degradation >> >> and you can help identify where our configurations differ, it could >> >> help fix >> >> the problem which would be awesome. >> > >> > -- >> > Radosław Szkodziński > >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
I compiled Xen 4.1.2 and 4.1.3 during a variety of GPU tests and saw the same problems while using the xl toolstack. I never had success with passthrough using xm, but I read plenty of guides about it. I had no luck using the packaged versions of Xen for a variety of reasons, and since I went down the path of compiling from source I didn''t bother looking back. I am working on a UEFI Booted GPT boot disk OCZ Vertex 3 SSD Debian Wheezy with Xen 4.2, so unstable everything essentially, yet my system has been rock solid for two months. Just last weekend I upgraded Debian and rebuilt the latest Xen, a lot of bugfixes to the Xen source since the last compile (the efi memory problem I encountered was finally fixed). Xen 4.2 has updates every week, and the xl toolstack is constantly under improvement ( http://blog.xen.org/index.php/2012/05/22/libxl-event-api-improvements/). I wouldn''t say that xl is stable yet though, when they give us a release date for Xen 4.2 stable then you may have a better answer. I can say that the xm toolstack was deprecated as of Xen 4.1, and is no longer being actively maintained ( http://wiki.xen.org/wiki/Choice_of_Toolstacks). It doesn''t compile by default with Xen 4.2, I don''t know if it would succeed if you tried given the changes to Xen''s source either. I have PVHVM for Windows and Linux, but I have not successfully passed the card to a Linux DomU yet (my own lack of linux experience and time). It works fine. I know stubdom is compiled when I build Xen, but I don''t know if it is put to use automatically; I have done no reading on this topic. The only other topic I can think of is qemu versions. Currently qemu-traditional is required for pci passthrough, though from testing the upstream qemu package has a lot of great features I would like to see. My understanding is that PCI Passthrough is not part of the standard qemu package, and adding it at this stage would be a lot of work, so they will probably update the qemu package as or after Xen 4.2 stable is released. The emulated graphics and disk IO were noticeably better with the upstream qemu version, so I really hope to see it implemented sooner than later. My conclusion agrees with yours, provided that unstable qemu-traditional is the same version used by Xen 4.1.2, the toolstack is one of the most likely places for these bugs to exist. On Tue, Jun 26, 2012 at 6:57 PM, Matthias < matthias.kannenberg@googlemail.com> wrote:> Well, then as i said before: I thing xm is a lot more stable right now > then xl and i don''t really see any benefit of using xl right now. > > Actually I tried using xl instead of xm last week and but run intu > issues with creating vifs when using nat. Interesting to see that > there are also issues with vga passthrough. > > Are there - right now - any benefits of using xl instead of xm? I > thought that it''s easier to run stubdoms with xl, but after reading > that stubdoms don''t bring you any significant performance increate > when using PVOPS-HVMs, i gave up on trieing to get stubdoms running. > So it would be nice to here if migrating to xl is actually something > to look into in the (near) future. > > 2012/6/27 Casey DeLorme <cdelorme@gmail.com>: > > I am using the xl toolstack, with Xen 4.2. That''s probably the > difference, > > I tried the pci flags (both inline and stand-alone versions of them) > without > > any changes. > > > > On Tue, Jun 26, 2012 at 7:19 AM, Matthias > > <matthias.kannenberg@googlemail.com> wrote: > >> > >> Hi, > >> > >> did some testing in my lunch break: > >> > >> # dmesg -c && xm dmesg -c > >> # xm destroy WINDOMU > >> # dmesg > >> [257342.161103] xen1: port 1(work) entered disabled state > >> [257342.162777] xen1: port 1(work) entered disabled state > >> [257345.003372] irq 18: nobody cared (try booting with the "irqpoll" > >> option) > >> [257345.004847] Pid: 11048, comm: qemu-dm Not tainted 3.4.2-xen #2 > >> [257345.006208] Call Trace: > >> [257345.006208] [<ffffffff800085d5>] dump_trace+0x85/0x1c0 > >> [257345.006208] [<ffffffff8088c036>] dump_stack+0x69/0x6f > >> [257345.006208] [<ffffffff800c9936>] __report_bad_irq+0x36/0xe0 > >> [257345.006208] [<ffffffff800c9c8b>] note_interrupt+0x1fb/0x240 > >> [257345.006208] [<ffffffff800c72c4>] handle_irq_event_percpu+0x94/0x1d0 > >> [257345.006208] [<ffffffff800c7464>] handle_irq_event+0x64/0x90 > >> [257345.006208] [<ffffffff800ca7b4>] handle_fasteoi_irq+0x64/0x120 > >> [257345.006208] [<ffffffff80008468>] handle_irq+0x18/0x30 > >> [257345.006208] [<ffffffff805c1cc4>] evtchn_do_upcall+0x1c4/0x2e0 > >> [257345.006208] [<ffffffff808a308e>] do_hypervisor_callback+0x1e/0x30 > >> [257345.006208] [<ffffffff8014802a>] fget_light+0x3a/0xc0 > >> [257345.006208] [<ffffffff80159c65>] do_select+0x315/0x660 > >> [257345.006208] [<ffffffff8015a15a>] core_sys_select+0x1aa/0x2e0 > >> [257345.006208] [<ffffffff8015a347>] sys_select+0xb7/0x110 > >> [257345.006208] [<ffffffff808a29bb>] system_call_fastpath+0x1a/0x1f > >> [257345.006208] [<00007f1a6d0081d3>] 0x7f1a6d0081d2 > >> [257345.006208] handlers: > >> [257345.006208] [<ffffffff80658a80>] usb_hcd_irq > >> [257345.006208] [<ffffffff80658a80>] usb_hcd_irq > >> [257345.006208] [<ffffffff80658a80>] usb_hcd_irq > >> [257345.006208] Disabling IRQ #18 > >> # xm dmesg > >> <nothing here> > >> # xm create WINDOMU > >> booted without problems and i played some Diablo 3 for testing > >> purpose: no performance change, everything normal > >> > >> Then i analysed the kernel output in dmesg from the kill and found out > >> that the IRQ 18 is not in fact my graphic card what i always thought, > >> but my usb host controler which i have forwarded to the domU, too. So > >> appereantly there is no nothing vga related in the dmesg. Then I > >> thought okay, if the dom0 in fact doesn''t care about the vga state, it > >> might actually be the domU and the only thing i guess can be > >> responsible for that is this in my domU config: > >> > >> ####################### > >> # PCI Power Management: > >> # > >> # If it''s set, the guest OS will be able to program D0-D3hot states of > >> the > >> # PCI device for the purpose of low power consumption. > >> # > >> pci_power_mgmt=1 > >> ####################### > >> > >> My new theory is that this allowes the domU (and therefor my windows) > >> to reset the vga on boot so i don''t have the problems you have.. > >> > >> Question is: Are you using this option in your domU or might that be > >> the difference in our configs? > >> > >> > >> > >> > >> 2012/6/26 Radoslaw Szkodzinski <astralstorm@gmail.com>: > >> > On Tue, Jun 26, 2012 at 2:51 AM, Casey DeLorme <cdelorme@gmail.com> > >> > wrote: > >> >> I agree with you that Xen has an awareness, but what I read suggested > >> >> that > >> >> the DomU is supposed to be responsible for the reset. > >> > > >> > Quite silly, many OSes and drivers don''t care about device shutdown on > >> > "poweroff". > >> > Why should they, the power usually will be off real soon anyway. > >> > I think currently only Linux cares enough - due to the need to support > >> > kexec. > >> > > >> > Suspend to ram is different matter. Perhaps that avenue would be > >> > useful to explore. > >> > (Attempting to S2R the VM instead of shutdown...) > >> > > >> > Still, it''d be nice for Xen to force reset the devices (FLR, then > >> > D0->D3->D0 ASPM, then finally PCI bus reset as a last resort) when the > >> > VM stops using the devices. > >> > Better safe than sorry. > >> > > >> >> In any event, please > >> >> do post your results. If you don''t have the same performance > >> >> degradation > >> >> and you can help identify where our configurations differ, it could > >> >> help fix > >> >> the problem which would be awesome. > >> > > >> > -- > >> > Radosław Szkodziński > > > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
The configuration file below resolved my problems. I''m passing through gfx device to Win7 domU (as a secondary device). Interestingly I can reboot the domU without any gfx issues. I''m also now running a custom kernel with pciback built-in, but I don''t think that''s the solution. I''ll do more troubleshooting soon to identify the culprit and share for posterity. Thanks all for the useful discussion and sustained interest -Chris On 6/25/12 3:51 PM, jocelyn falempe wrote:> Hi, > > I''ve got a similar setup : > > core i7-3770 + ASRock z77 pro 4M > and a sapphire radeon HD7950 > > Ubuntu 12.04 as Dom0, and Win8 release preview as HVM guest. > > I can start my VM multiple time without issue. > as I have 3 USB controller and 2 SATA controller, I use pci passthrough > to pass USB and SATA devices too > so my razer mouse, and secondary sata drive is "native" on windows too. > > here is my script (copied from someone) to reserve the PCI devices for > my HVM : > > remove_device () { > BDF=$1 > # Unbind a PCI function from its driver as necessary > [ ! -e /sys/bus/pci/devices/$BDF/driver/unbind ] || \ > echo -n $BDF > /sys/bus/pci/devices/$BDF/driver/unbind > # Add a new slot to the PCI Backend''s list > echo -n $BDF > /sys/bus/pci/drivers/pciback/new_slot > # Now that the backend is watching for the slot, bind to it > echo -n $BDF > /sys/bus/pci/drivers/pciback/bind > } > > #USB Controller : > remove_device "0000:00:1a.0" > #SATA Controller : > remove_device "0000:04:00.0" > #Radeon 7950 > remove_device "0000:01:00.0" > #Radeon 7950 audio > remove_device "0000:01:00.1" > > and here is my .cfg file to launch the VM (xm create win8_lan.cfg) : > cat win8_lan.cfg > kernel="/usr/lib/xen-default/boot/hvmloader" > builder=''hvm'' > memory = 8096 > vcpus=6 > name = "win8_new" > vif = [ ''type=ioemu, bridge=br0'' ] > disk = [''file:/xen-images/win8_2.img,ioemu:hda,w''] > acpi = 1 > boot="c" > sdl=0 > serial=''pty'' > vnc=1 > pci=[ ''00:1a.0'', ''01:00.0'', ''01:00.1'', ''04:00.0'' ] > > hope it helps > > Regards, > > Jocelyn > > > On 06/25/2012 05:32 PM, Radoslaw Szkodzinski wrote: >> On Mon, Jun 25, 2012 at 5:21 PM, Matthias >> <matthias.kannenberg@googlemail.com> wrote: >>> Maybe you should give xm a try just to see if it does the trick. I >>> never got vga passthrough working with xl (and from my understanding, >>> it''s a lot mor complicated there with compiling the vga bios into xen >>> and manual calculating vga adress ranges.. with xm, I''m doing neither >>> of it). >> Why? I''m not using the VGA Passthrough, The card is set up as a >> secondary. >> That shouldn''t need any VGA BIOS. Also, it works fine for the first >> boot very well! >> >> The problem occurs on the second start of the same VM. >> >>> Also, do you increase the log level for xen? my kernel line is: >>> multiboot /boot/xen.gz dom0_mem=2048M iommu=1 loglvl=all >>> guest_loglvl=all >> Will do. (except dom0_mem) >> >>> What kernel are you using? If you want i can provide my build commands >>> for the xen-patched openSuse Kernel.. >> I don''t want to use the XenLinux kernel. PVOps only please. >> Unless the Xen kernel is actually Pvops-based, in which case why would >> I want to use OpenSUSE one instead of vanilla? >> >> As I''ve mentioned, vanilla 3.4.1. >> > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users