I''ve just had my first attempt at getting VGA passthrough to work, and it crashed the machine. I''m trying to understand whether this is a problem with my hardware, configuration or a software problem. I''m running Xen 4.1.2 with Linux 3.1.2 The CPU is an Intel Core i5-650 http://ark.intel.com/products/43546/Intel-Core-i5-650-Processor-%284M-Cache-3_20-GHz%29 The Video Card is and Intel Core Processor Integrated Graphics Controller PCI ID: 8086 0042 I have hidden the vga device with xen-pciback.hide=(00:02.0) on the kernel command line The xen config file is: name = ''11-1'' kernel = ''/usr/lib/xen/boot/hvmloader'' builder = ''hvm'' device_model = ''/usr/lib/xen/bin/qemu-dm'' boot = ''c'' serial = ''pty'' usbdevice = ''tablet'' disk = [ ''phy:/dev/Master/Root-11-0001,hda,w'' ] memory = 1000 vif = [ ''bridge=br-internal,mac=04:fb:7a:22:e8:4a'',''bridge=br-dmz,mac=04:fb:7a:22:e8:4b'',''bridge=br-local,mac=04:fb:7a:22:e8:4c'' ] vcpus = 1 on_poweroff = ''destroy'' on_reboot = ''destroy'' on_crash = ''destroy'' gfx_passthru=1 pci=[''00:02.0''] When I try to start the DomU I get the following output, I get the prompt back and then the system reboots. Parsing config file xen-config xc: info: VIRTUAL MEMORY ARRANGEMENT: Loader: 0000000000100000->00000000001795d0 TOTAL: 0000000000000000->000000003e000000 ENTRY ADDRESS: 0000000000100000 xc: info: PHYSICAL MEMORY ALLOCATION: 4KB PAGES: 0x0000000000000200 2MB PAGES: 0x00000000000001ef 1GB PAGES: 0x0000000000000000 libxl: error: libxl_pci.c:712:do_pci_add xc_assign_device failed Daemon running with PID 2050
On Thu, Dec 08, 2011 at 12:00:07PM +0000, Anthony Wright wrote:> I''ve just had my first attempt at getting VGA passthrough to work, and > it crashed the machine. I''m trying to understand whether this is a > problem with my hardware, configuration or a software problem. > > I''m running Xen 4.1.2 with Linux 3.1.2 > > > The CPU is an Intel Core i5-650 > http://ark.intel.com/products/43546/Intel-Core-i5-650-Processor-%284M-Cache-3_20-GHz%29And what is your motherboard? Does it have VT-d?> > The Video Card is and Intel Core Processor Integrated Graphics Controller > PCI ID: 8086 0042 > > > I have hidden the vga device with xen-pciback.hide=(00:02.0) on the > kernel command line > > The xen config file is: > name = ''11-1'' > kernel = ''/usr/lib/xen/boot/hvmloader'' > builder = ''hvm'' > device_model = ''/usr/lib/xen/bin/qemu-dm'' > boot = ''c'' > serial = ''pty'' > usbdevice = ''tablet'' > disk = [ ''phy:/dev/Master/Root-11-0001,hda,w'' ] > memory = 1000 > vif = [ > ''bridge=br-internal,mac=04:fb:7a:22:e8:4a'',''bridge=br-dmz,mac=04:fb:7a:22:e8:4b'',''bridge=br-local,mac=04:fb:7a:22:e8:4c'' > ] > vcpus = 1 > on_poweroff = ''destroy'' > on_reboot = ''destroy'' > on_crash = ''destroy'' > > gfx_passthru=1 > pci=[''00:02.0''] > > When I try to start the DomU I get the following output, I get the > prompt back and then the system reboots. > > Parsing config file xen-config > xc: info: VIRTUAL MEMORY ARRANGEMENT: > Loader: 0000000000100000->00000000001795d0 > TOTAL: 0000000000000000->000000003e000000 > ENTRY ADDRESS: 0000000000100000 > xc: info: PHYSICAL MEMORY ALLOCATION: > 4KB PAGES: 0x0000000000000200 > 2MB PAGES: 0x00000000000001ef > 1GB PAGES: 0x0000000000000000 > libxl: error: libxl_pci.c:712:do_pci_add xc_assign_device failed > Daemon running with PID 2050 > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel
On 14/12/2011 21:38, Konrad Rzeszutek Wilk wrote:> On Thu, Dec 08, 2011 at 12:00:07PM +0000, Anthony Wright wrote: >> I''ve just had my first attempt at getting VGA passthrough to work, and >> it crashed the machine. I''m trying to understand whether this is a >> problem with my hardware, configuration or a software problem. >> >> I''m running Xen 4.1.2 with Linux 3.1.2 >> >> >> The CPU is an Intel Core i5-650 >> http://ark.intel.com/products/43546/Intel-Core-i5-650-Processor-%284M-Cache-3_20-GHz%29 > And what is your motherboard? Does it have VT-d?That was the important question... I''d missed the BIOS option to enable VT-d which was disabled by default. I enabled the BIOS option and everything worked, and it''s very cool. :-) This does beg a couple of other questions though.... Shouldn''t I get a nice(ish) error message if I try to start a VGA passthrough VM on hardware that doesn''t support it rather than crashing the machine? Can I tell whether hardware will support VGA passthrough without trying to start a VM that uses it? Also when I shut down the VM I have to use ''xl destroy'' as I''m testing with Windows. After issuing the ''xl destroy'' there is a long pause and then the error: libxl: error: libxl_device.c:470:libxl__wait_for_device_model Device Model not ready libxl: error: libxl_pci.c:892:do_pci_remove Device Model didn''t respond in time If I pass through more than one device, I get the pause and the message for each device.
On Thu, Dec 15, 2011 at 03:54:00PM +0000, Anthony Wright wrote:> On 14/12/2011 21:38, Konrad Rzeszutek Wilk wrote: > > On Thu, Dec 08, 2011 at 12:00:07PM +0000, Anthony Wright wrote: > >> I''ve just had my first attempt at getting VGA passthrough to work, and > >> it crashed the machine. I''m trying to understand whether this is a > >> problem with my hardware, configuration or a software problem. > >> > >> I''m running Xen 4.1.2 with Linux 3.1.2 > >> > >> > >> The CPU is an Intel Core i5-650 > >> http://ark.intel.com/products/43546/Intel-Core-i5-650-Processor-%284M-Cache-3_20-GHz%29 > > And what is your motherboard? Does it have VT-d? > That was the important question... I''d missed the BIOS option to enable > VT-d which was disabled by default. I enabled the BIOS option and > everything worked, and it''s very cool. :-) > > This does beg a couple of other questions though.... > > Shouldn''t I get a nice(ish) error message if I try to start a VGA > passthrough VM on hardware that doesn''t support it rather than crashing > the machine?There is this http://wiki.xen.org/xenwiki/VTdHowTo which gives you a nice idea of what works.> > Can I tell whether hardware will support VGA passthrough without trying > to start a VM that uses it? > > Also when I shut down the VM I have to use ''xl destroy'' as I''m testing > with Windows. After issuing the ''xl destroy'' there is a long pause and > then the error: > > libxl: error: libxl_device.c:470:libxl__wait_for_device_model Device > Model not ready > libxl: error: libxl_pci.c:892:do_pci_remove Device Model didn''t > respond in time > > If I pass through more than one device, I get the pause and the message > for each device.Hm, it looks to be sending ''pci-rem'' to the XenStore, but I am not sure if QEMU understands it. Did you compile Xen 4.1.2 by yourself? Did it compile with CONFIG_PASSTHROUGH? You should see some messages in the qemu.log about hot-remove and such.
On 15/12/2011 21:12, Konrad Rzeszutek Wilk wrote:> On Thu, Dec 15, 2011 at 03:54:00PM +0000, Anthony Wright wrote: >> On 14/12/2011 21:38, Konrad Rzeszutek Wilk wrote: >>> On Thu, Dec 08, 2011 at 12:00:07PM +0000, Anthony Wright wrote: >>>> I''ve just had my first attempt at getting VGA passthrough to work, and >>>> it crashed the machine. I''m trying to understand whether this is a >>>> problem with my hardware, configuration or a software problem. >>>> >>>> I''m running Xen 4.1.2 with Linux 3.1.2 >>>> >>>> >>>> The CPU is an Intel Core i5-650 >>>> http://ark.intel.com/products/43546/Intel-Core-i5-650-Processor-%284M-Cache-3_20-GHz%29 >>> And what is your motherboard? Does it have VT-d? >> That was the important question... I''d missed the BIOS option to enable >> VT-d which was disabled by default. I enabled the BIOS option and >> everything worked, and it''s very cool. :-) >> >> This does beg a couple of other questions though.... >> >> Shouldn''t I get a nice(ish) error message if I try to start a VGA >> passthrough VM on hardware that doesn''t support it rather than crashing >> the machine? > > There is this http://wiki.xen.org/xenwiki/VTdHowTo which gives you a > nice idea of what works.Unfortunately I spoke a little too soon. I''m running with a Windows 7 Pro, 32 bit iso image, to install a new version of windows into the VM. It seems to start nicely and I get a pulsing windows logo on the screen, but it never progresses to the next page, and the VM consumes a huge amount of CPU. If I run it purely in a VNC session, everything runs smoothly and quickly. We noticed that there are a lot of VT-d features disabled, and wondered if this might be the cause. The motherboard is an intel DQ57TM, and the processor is an Intel Core i5-650, I have enabled VT-d and FLR within the motherboard which seem to be the only relevant motherboard switches. Inside xl dmesg all the additional VT-d features are not enabled (Snoop Control, Dom0 DMA Passthrough, Queued Invalidation, Interrupt Remapping, Shared EPT tables) and I wondered if this was the problem. I tried starting with iommu=1 on the xen command line and without it. The output from xl dmesg is very similar and the VM behaves the same in both cases, but I get a number of iommu.c errors messages when the ''iommu=1'' parameter is present. In both cases Xen also outputs a number of mm.c errors which is similar to a problem I reported a while ago. I have attach xl dmesg logs from both cases.>> Can I tell whether hardware will support VGA passthrough without trying >> to start a VM that uses it? >> >> Also when I shut down the VM I have to use ''xl destroy'' as I''m testing >> with Windows. After issuing the ''xl destroy'' there is a long pause and >> then the error: >> >> libxl: error: libxl_device.c:470:libxl__wait_for_device_model Device >> Model not ready >> libxl: error: libxl_pci.c:892:do_pci_remove Device Model didn''t >> respond in time >> >> If I pass through more than one device, I get the pause and the message >> for each device. > Hm, it looks to be sending ''pci-rem'' to the XenStore, but I am not sure > if QEMU understands it. Did you compile Xen 4.1.2 by yourself? Did it > compile with CONFIG_PASSTHROUGH? > > You should see some messages in the qemu.log about hot-remove and such.Yes I compiled Xen 4.1.2 by myself, but I don''t know is CONFIG_PASSTHROUGH is enabled or not - there''s no mention of it in the build logs. I see a hot-remove entry in the qemu log, and have attached the whole log as it''s not very long. Anthony. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, Dec 19, 2011 at 03:32:51PM +0000, Anthony Wright wrote:> On 15/12/2011 21:12, Konrad Rzeszutek Wilk wrote: > > On Thu, Dec 15, 2011 at 03:54:00PM +0000, Anthony Wright wrote: > >> On 14/12/2011 21:38, Konrad Rzeszutek Wilk wrote: > >>> On Thu, Dec 08, 2011 at 12:00:07PM +0000, Anthony Wright wrote: > >>>> I''ve just had my first attempt at getting VGA passthrough to work, and > >>>> it crashed the machine. I''m trying to understand whether this is a > >>>> problem with my hardware, configuration or a software problem. > >>>> > >>>> I''m running Xen 4.1.2 with Linux 3.1.2 > >>>> > >>>> > >>>> The CPU is an Intel Core i5-650 > >>>> http://ark.intel.com/products/43546/Intel-Core-i5-650-Processor-%284M-Cache-3_20-GHz%29 > >>> And what is your motherboard? Does it have VT-d? > >> That was the important question... I''d missed the BIOS option to enable > >> VT-d which was disabled by default. I enabled the BIOS option and > >> everything worked, and it''s very cool. :-) > >> > >> This does beg a couple of other questions though.... > >> > >> Shouldn''t I get a nice(ish) error message if I try to start a VGA > >> passthrough VM on hardware that doesn''t support it rather than crashing > >> the machine? > > > > There is this http://wiki.xen.org/xenwiki/VTdHowTo which gives you a > > nice idea of what works. > Unfortunately I spoke a little too soon. I''m running with a Windows 7 > Pro, 32 bit iso image, to install a new version of windows into the VM. > It seems to start nicely and I get a pulsing windows logo on the screen, > but it never progresses to the next page, and the VM consumes a huge > amount of CPU. If I run it purely in a VNC session, everything runs > smoothly and quickly. We noticed that there are a lot of VT-d features > disabled, and wondered if this might be the cause. The motherboard is an > intel DQ57TM, and the processor is an Intel Core i5-650, I have enabled > VT-d and FLR within the motherboard which seem to be the only relevant > motherboard switches.Are you enabling gfx_passthrough or using the VNC _and_ the PCI passthrough features (which is what I was using this last week). Something like this: builder=''hvm'' memory = 2048 name = "Windows7" vcpus=3 disk = [ ''phy:/dev/vg_guest/Win7_Home,hda,w''] vnc=1 videoram=8 vnclisten="0.0.0.0" vncpasswd='''' #stdvga=0 serial=''pty'' tsc_mode=0 usb=1 usb_add="host:045e:0040" usbdevice=''tablet'' xen_platform_pci=1 # Remember QEMU_AUDIO_DRV=pa soundhw=''es1370'' pci=[''01:00.0'',''01:00.1'',''02:00.0''] Where the PCI card is: 01:00.0 VGA compatible controller: ATI Technologies Inc RV770 [Radeon HD 4870] 01:00.1 Audio device: ATI Technologies Inc HD48x0 audio Or are you passing in an Nvidia card? (The motherboard I am using is an AMD one).> > Inside xl dmesg all the additional VT-d features are not enabled (Snoop > Control, Dom0 DMA Passthrough, Queued Invalidation, Interrupt Remapping, > Shared EPT tables) and I wondered if this was the problem. I triedIf it was I would think you would get tons of warnings from the Xen hypervisor about failed mappings and such. [edit: and it seems you are getting faults]> starting with iommu=1 on the xen command line and without it. The output > from xl dmesg is very similar and the VM behaves the same in both cases, > but I get a number of iommu.c errors messages when the ''iommu=1'' > parameter is present. In both cases Xen also outputs a number of mm.c > errors which is similar to a problem I reported a while ago. I have > attach xl dmesg logs from both cases.You mean these:> (XEN) [VT-D]iommu.c:853: iommu_fault_status: Fault Overflow > (XEN) [VT-D]iommu.c:856: iommu_fault_status: Primary Pending Fault > (XEN) [VT-D]iommu.c:831: DMAR:[DMA Write] Request device [00:02.0] fault addr 400000000, iommu reg = fff16000 > (XEN) DMAR:[fault reason 05h] PTE Write access is not setHm, and 0000:00:02.0 is your VGA card right? These ones you can ignore:> (XEN) mm.c:907:d0 Error getting mfn 2409c (pfn 55555555) from L1 entry 000000002409c023 for l1e_owner=0, pg_owner=0 > (XEN) mm.c:907:d0 Error getting mfn 2409d (pfn 55555555) from L1 entry 000000002409d023 for l1e_owner=0, pg_owner=0 > (XEN) mm.c:907:d0 Error getting mfn 2409e (pfn 55555555) from L1 entry 000000002409e023 for l1e_owner=0, pg_owner=0 > (XEN) mm.c:907:d0 Error getting mfn 2409f (pfn 55555555) from L1 entry 000000002409f023 for l1e_owner=0, pg_owner=0