Xen users - Sep 2013 - VGA Passthrough of AMD FirePro W7000 to Windows 8: Not enough resources

I''m using Xen 4.3, installed from source on Ubuntu 13.04. I''ve
successfully installed a Windows 8 guest, and had Xen pass it an AMD 
FirePro W7000 GPU (technically 2 PCI devices - the second is for the 
sound), as well as the virtualised graphics adapter. Windows finds the 
card fine, and I''ve installed AMD''s drivers. Device Manager
says that
the card cannot find enough resources (code 12). I''ve checked the IO, 
IRQ, and Memory details in DM, and there don''t seem to be any clashes.

I did try disabling the virtualised card from Windows, but it seemed to 
keep using it anyway, even after a restart. It didn''t make any 
difference to the FirePro in any case.

I''d be grateful for some help.

Many thanks,

Toby

On Sun, 29 Sep 2013 15:01:01 +0100, Toby Miller <tobycmiller@gmail.com> 
 wrote:
> I''m using Xen 4.3, installed from source on Ubuntu 13.04.
I''ve
> successfully installed a Windows 8 guest, and had Xen pass it an AMD
> FirePro W7000 GPU (technically 2 PCI devices - the second is for the
> sound), as well as the virtualised graphics adapter. Windows finds 
> the
> card fine, and I''ve installed AMD''s drivers. Device
Manager says that
> the card cannot find enough resources (code 12). I''ve checked the
IO,
> IRQ, and Memory details in DM, and there don''t seem to be any 
> clashes.
>
> I did try disabling the virtualised card from Windows, but it seemed
> to keep using it anyway, even after a restart. It didn''t make any
> difference to the FirePro in any case.
>
> I''d be grateful for some help.
 I haven''t tried Windows 8, but on XP64 disabling the Cirrus card in
 device manager fixed the problem for me with Nvidia card passthrough.

 You could try setting gfx_passthru=1, which should disable the
 Cirrus emulation alltogether. That may make Windows domU completely
 fail to boot, though.

 Did you do a clean Win8 install and reboot the host fully? Is the
 FirePro the primary GPU on the host? Or is it completely untouched
 in dom0 (no dom0 or BIOS output on it)?

 Gordan

On Sep 30, 2013, at 8:26 AM, Toby Miller <tobycmiller@gmail.com> wrote:

I''m using Xen 4.3, installed from source on Ubuntu 13.04. I''ve
successfully
installed a Windows 8 guest, and had Xen pass it an AMD FirePro W7000 GPU
(technically 2 PCI devices - the second is for the sound), as well as the
virtualised graphics adapter. Windows finds the card fine, and I''ve
installed AMD''s drivers. Device Manager says that the card cannot find
enough resources (code 12). I''ve checked the IO, IRQ, and Memory
details in
DM, and there don''t seem to be any clashes.

I did try disabling the virtualised card from Windows, but it seemed to
keep using it anyway, even after a restart. It didn''t make any
difference
to the FirePro in any case.

I''d be grateful for some help.

Many thanks,

Toby


Hello Toby,

I can''t say whether or not it will help at all, but if your problem
relates
to IRQ issues, it may help to try turning on/off MSI translation for the
device, or possibly changing its virtual slot assignment.

The syntax for changing these options in your domain''s cfg file can be
found here: http://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html#devices

Cheers,
Andrew


_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

>> gordan@bobich.net wrote:
>   On Sun, 29 Sep 2013 15:01:01 +0100, Toby
Miller<tobycmiller@gmail.com>
>   wrote:
>> >I''m using Xen 4.3, installed from source on Ubuntu 13.04.
I''ve
>> >successfully installed a Windows 8 guest, and had Xen pass it an
AMD
>> >FirePro W7000 GPU (technically 2 PCI devices - the second is for
the
>> >sound), as well as the virtualised graphics adapter. Windows finds
>> >the
>> >card fine, and I''ve installed AMD''s drivers.
Device Manager says that
>> >the card cannot find enough resources (code 12). I''ve
checked the IO,
>> >IRQ, and Memory details in DM, and there don''t seem to be
any
>> >clashes.
>> >
>> >I did try disabling the virtualised card from Windows, but it
seemed
>> >to keep using it anyway, even after a restart. It didn''t
make any
>> >difference to the FirePro in any case.
>> >
>> >I''d be grateful for some help.
>   I haven''t tried Windows 8, but on XP64 disabling the Cirrus card
in
>   device manager fixed the problem for me with Nvidia card passthrough.
>
>   You could try setting gfx_passthru=1, which should disable the
>   Cirrus emulation alltogether. That may make Windows domU completely
>   fail to boot, though.
>
>   Did you do a clean Win8 install and reboot the host fully? Is the
>   FirePro the primary GPU on the host? Or is it completely untouched
>   in dom0 (no dom0 or BIOS output on it)?
>
>   Gordan
Thanks for your reply. It turns out the problem was that I''d forgotten 
to put device_model_version="qemu-xen-traditional", so I was using a 
qemu that didn''t support gfx_passthrough. The FirePro is not used in 
dom0 at all. I''m using pciback to hide it.

I now have another problem though. The Ubuntu host crashes a lot when 
the Windows 8 guest is running. Mostly it happens when I try to shut 
down Windows - I think it might be a result of the FirePro card being 
released -, but sometimes it happens randomly too. The screen completely 
freezes, and no ssh or anything. I should mention that I''m not passing 
through the audio pci device for the FirePro, although it is bound to 
pciback so the host can''t be using it either. If I try to pass that 
through it never actually boots. I''ve had a lot of blue screens, and a 
lot of host crashes. I''m passing through a USB device too (i.e. the PCI
USB device), and that works fine.

Googling doesn''t seem to come up with much for host crashes. What can I
do to diagnose the problem? There''s nothing in the syslog.

Thanks a lot!

Toby

On 10/02/2013 01:58 PM, Toby Miller wrote:
>  >> gordan@bobich.net wrote:
>>   On Sun, 29 Sep 2013 15:01:01 +0100, Toby
>> Miller<tobycmiller@gmail.com>   wrote:
>>> >I''m using Xen 4.3, installed from source on Ubuntu
13.04. I''ve
>>> >successfully installed a Windows 8 guest, and had Xen pass it
an AMD
>>> >FirePro W7000 GPU (technically 2 PCI devices - the second is
for the
>>> >sound), as well as the virtualised graphics adapter. Windows
finds
>>> >the
>>> >card fine, and I''ve installed AMD''s drivers.
Device Manager says that
>>> >the card cannot find enough resources (code 12). I''ve
checked the IO,
>>> >IRQ, and Memory details in DM, and there don''t seem to
be any
>>> >clashes.
>>> >
>>> >I did try disabling the virtualised card from Windows, but it
seemed
>>> >to keep using it anyway, even after a restart. It
didn''t make any
>>> >difference to the FirePro in any case.
>>> >
>>> >I''d be grateful for some help.
>>   I haven''t tried Windows 8, but on XP64 disabling the Cirrus
card in
>>   device manager fixed the problem for me with Nvidia card passthrough.
>>
>>   You could try setting gfx_passthru=1, which should disable the
>>   Cirrus emulation alltogether. That may make Windows domU completely
>>   fail to boot, though.
>>
>>   Did you do a clean Win8 install and reboot the host fully? Is the
>>   FirePro the primary GPU on the host? Or is it completely untouched
>>   in dom0 (no dom0 or BIOS output on it)?
>>
>>   Gordan
> Thanks for your reply. It turns out the problem was that I''d
forgotten
> to put device_model_version="qemu-xen-traditional", so I was
using a
> qemu that didn''t support gfx_passthrough. The FirePro is not used
in
> dom0 at all. I''m using pciback to hide it.
>
> I now have another problem though. The Ubuntu host crashes a lot when
> the Windows 8 guest is running. Mostly it happens when I try to shut
> down Windows - I think it might be a result of the FirePro card being
> released -, but sometimes it happens randomly too. The screen completely
> freezes, and no ssh or anything. I should mention that I''m not
passing
> through the audio pci device for the FirePro, although it is bound to
> pciback so the host can''t be using it either. If I try to pass
that
> through it never actually boots. I''ve had a lot of blue screens,
and a
> lot of host crashes. I''m passing through a USB device too (i.e.
the PCI
> USB device), and that works fine.
>
> Googling doesn''t seem to come up with much for host crashes. What
can I
> do to diagnose the problem? There''s nothing in the syslog.
Can you try it with domU limited to 1GB of RAM? Also, can you add your 
physical e820 memory map as reported in dmesg?

It is plausible you might be running into the same issue I had with domU 
memory overwriting the physical IOMEM without getting remapped by the IOMMU.

Does your motherboard feature NF200 PCIe bridges?

Gordan

On 02/10/13 01:05, Andrew Bobulsky wrote:
> On Sep 30, 2013, at 8:26 AM, Toby Miller <tobycmiller@gmail.com 
> <mailto:tobycmiller@gmail.com>> wrote:
>
>> I''m using Xen 4.3, installed from source on Ubuntu 13.04.
I''ve
>> successfully installed a Windows 8 guest, and had Xen pass it an AMD 
>> FirePro W7000 GPU (technically 2 PCI devices - the second is for the 
>> sound), as well as the virtualised graphics adapter. Windows finds 
>> the card fine, and I''ve installed AMD''s drivers.
Device Manager says
>> that the card cannot find enough resources (code 12). I''ve
checked
>> the IO, IRQ, and Memory details in DM, and there don''t seem to
be any
>> clashes.
>>
>> I did try disabling the virtualised card from Windows, but it seemed 
>> to keep using it anyway, even after a restart. It didn''t make
any
>> difference to the FirePro in any case.
>>
>> I''d be grateful for some help.
>>
>> Many thanks,
>>
>> Toby
>
> Hello Toby,
>
> I can''t say whether or not it will help at all, but if your
problem
> relates to IRQ issues, it may help to try turning on/off MSI 
> translation for the device, or possibly changing its virtual slot 
> assignment.
>
> The syntax for changing these options in your domain''s cfg file
can be
> found here: http://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html#devices
>
> Cheers,
> Andrew
Thanks for your help. I''ve managed to get it booting now, although the 
host crashes all the time. I''ve tried turning on MSI translation but 
that doesn''t seem to make much diference. Details are in my reply to 
gordan@bobich.net.

Thanks!

Toby



_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

On 10/02/2013 02:14 PM, Toby Miller wrote:
>
> On 02/10/13 01:05, Andrew Bobulsky wrote:
>> On Sep 30, 2013, at 8:26 AM, Toby Miller <tobycmiller@gmail.com
>> <mailto:tobycmiller@gmail.com>> wrote:
>>
>>> I''m using Xen 4.3, installed from source on Ubuntu 13.04.
I''ve
>>> successfully installed a Windows 8 guest, and had Xen pass it an
AMD
>>> FirePro W7000 GPU (technically 2 PCI devices - the second is for
the
>>> sound), as well as the virtualised graphics adapter. Windows finds
>>> the card fine, and I''ve installed AMD''s drivers.
Device Manager says
>>> that the card cannot find enough resources (code 12). I''ve
checked
>>> the IO, IRQ, and Memory details in DM, and there don''t
seem to be any
>>> clashes.
>>>
>>> I did try disabling the virtualised card from Windows, but it
seemed
>>> to keep using it anyway, even after a restart. It didn''t
make any
>>> difference to the FirePro in any case.
>>>
>>> I''d be grateful for some help.
>>>
>>> Many thanks,
>>>
>>> Toby
>>
>> Hello Toby,
>>
>> I can''t say whether or not it will help at all, but if your
problem
>> relates to IRQ issues, it may help to try turning on/off MSI
>> translation for the device, or possibly changing its virtual slot
>> assignment.
>>
>> The syntax for changing these options in your domain''s cfg
file can be
>> found here:
http://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html#devices
>>
>> Cheers,
>> Andrew
>
> Thanks for your help. I''ve managed to get it booting now, although
the
> host crashes all the time. I''ve tried turning on MSI translation
but
> that doesn''t seem to make much diference. Details are in my reply
to
> gordan@bobich.net.
1) Check lspci and make sure nothing is sharing the passthrough GPU 
interrupt.

2) You don''t need gfx_passthrough=1, I have my VMs running without it
as
secondary VGA passthrough. BIOS/boot output comes up on VNC, login 
screen and later come up on the GPU itself. Personally I have up on ATI 
cards, but my faux-Quadro cards work very well in this kind of a setup.

Gordan

> On 2 Oct 2013, at 14:06, Gordan Bobic <gordan@bobich.net> wrote:
>
>> On 10/02/2013 01:58 PM, Toby Miller wrote:
>> >> gordan@bobich.net wrote:
>>>  On Sun, 29 Sep 2013 15:01:01 +0100, Toby
>>> Miller<tobycmiller@gmail.com>   wrote:
>>>> >I''m using Xen 4.3, installed from source on Ubuntu
13.04. I''ve
>>>> >successfully installed a Windows 8 guest, and had Xen pass
it an AMD
>>>> >FirePro W7000 GPU (technically 2 PCI devices - the second
is for the
>>>> >sound), as well as the virtualised graphics adapter.
Windows finds
>>>> >the
>>>> >card fine, and I''ve installed AMD''s
drivers. Device Manager says that
>>>> >the card cannot find enough resources (code 12).
I''ve checked the IO,
>>>> >IRQ, and Memory details in DM, and there don''t
seem to be any
>>>> >clashes.
>>>> >
>>>> >I did try disabling the virtualised card from Windows, but
it seemed
>>>> >to keep using it anyway, even after a restart. It
didn''t make any
>>>> >difference to the FirePro in any case.
>>>> >
>>>> >I''d be grateful for some help.
>>>  I haven''t tried Windows 8, but on XP64 disabling the
Cirrus card in
>>>  device manager fixed the problem for me with Nvidia card
passthrough.
>>>
>>>  You could try setting gfx_passthru=1, which should disable the
>>>  Cirrus emulation alltogether. That may make Windows domU
completely
>>>  fail to boot, though.
>>>
>>>  Did you do a clean Win8 install and reboot the host fully? Is the
>>>  FirePro the primary GPU on the host? Or is it completely untouched
>>>  in dom0 (no dom0 or BIOS output on it)?
>>>
>>>  Gordan
>> Thanks for your reply. It turns out the problem was that I''d
forgotten
>> to put device_model_version="qemu-xen-traditional", so I was
using a
>> qemu that didn''t support gfx_passthrough. The FirePro is not
used in
>> dom0 at all. I''m using pciback to hide it.
>>
>> I now have another problem though. The Ubuntu host crashes a lot when
>> the Windows 8 guest is running. Mostly it happens when I try to shut
>> down Windows - I think it might be a result of the FirePro card being
>> released -, but sometimes it happens randomly too. The screen
completely
>> freezes, and no ssh or anything. I should mention that I''m not
passing
>> through the audio pci device for the FirePro, although it is bound to
>> pciback so the host can''t be using it either. If I try to pass
that
>> through it never actually boots. I''ve had a lot of blue
screens, and a
>> lot of host crashes. I''m passing through a USB device too
(i.e. the PCI
>> USB device), and that works fine.
>>
>> Googling doesn''t seem to come up with much for host crashes.
What can I
>> do to diagnose the problem? There''s nothing in the syslog.
>
> Can you try it with domU limited to 1GB of RAM? Also, can you add your
physical e820 memory map as reported in dmesg?
>
> It is plausible you might be running into the same issue I had with domU
memory overwriting the physical IOMEM without getting remapped by the IOMMU.
>
> Does your motherboard feature NF200 PCIe bridges?
>
> Gordan
Your overwriting memory suggestion sounds likely. I''ve tried with 1GiB
but the host still crashes a lot. The first time I ran it it booted,
and then shutdown fine, but starting it again crashed the host. The
second and third tries resulted in a host crash before Windows had
booted even once. Sometimes as the host crashes the mouse movement
slows down, before stopping altogether. Would this make sense if the
host''s memory were being overwritten?

Is this the memory map you wanted?:

[root@yeti /etc/xen]% dmesg | grep e820
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] e820: update [mem 0x00000000-0x0000ffff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] e820: last_pfn = 0x880000 max_arch_pfn = 0x400000000
[    0.000000] e820: last_pfn = 0x7d800 max_arch_pfn = 0x400000000
[    0.000000] e820: [mem 0x90000000-0xfed1bfff] available for PCI devices
[    8.827099] e820: reserve RAM buffer [mem 0x0009e000-0x0009ffff]
[    8.827101] e820: reserve RAM buffer [mem 0x7cdad000-0x7fffffff]
[    8.827103] e820: reserve RAM buffer [mem 0x7d334000-0x7fffffff]
[    8.827105] e820: reserve RAM buffer [mem 0x7d800000-0x7fffffff]

The motherboard is an EVGA SR-X, and I''m not sure how to tell whether
it has NF200 PCIe bridges.

Thanks!

Toby

AFAIR with Xen 4.3.0, Xen has to be patched with 

ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patchcd
/path/to/your/xen-4.3.0/directory

wget ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch -O - | patch
-p1Install xen (make & Co).

This patch below was sent to Xen mailing list a couple months ago. It should
prevent you from getting error code 12.

Download the following script to manage domU.

ftp://ftp.enjellic.com/pub/xen/run-passthrough

Update it to your own needs.

When installing AMD driver avoid to install Catalyst Center -- to avoid blue
screen.

I did it several time with my HD 7970 and it worked perfectly.

http://www.davidgis.fr/blog/index.php?2013/07/13/954-xen-430-stable-vga-passthrourh-for-ati
http://www.davidgis.fr/blog/index.php?2013/04/05/937-xen-43-unstable-vga-passthrough-hd-7970-windows-7-64-bits-with-more-than-3gb-for-ram





________________________________
 De : Toby Miller <tobycmiller@gmail.com>
À : xen-users@lists.xen.org; gordan@bobich.net 
Envoyé le : Mercredi 2 octobre 2013 14h58
Objet : Re: [Xen-users] VGA Passthrough of AMD FirePro W7000 to Windows 8: Not
enough resources
 
>> gordan@bobich.net wrote:
>   On Sun, 29 Sep 2013 15:01:01 +0100, Toby
Miller<tobycmiller@gmail.com> 
>   wrote:
>> >I''m using Xen 4.3, installed from source on Ubuntu 13.04.
I''ve
>> >successfully installed a Windows 8 guest, and had Xen pass it an
AMD
>> >FirePro W7000 GPU (technically 2 PCI devices - the second is for
the
>> >sound), as well as the virtualised graphics adapter. Windows finds
>> >the
>> >card fine, and I''ve installed AMD''s drivers.
Device Manager says that
>> >the card cannot find enough resources (code 12). I''ve
checked the IO,
>> >IRQ, and Memory details in DM, and there don''t seem to be
any
>> >clashes.
>> >
>> >I did try disabling the virtualised card from Windows, but it
seemed
>> >to keep using it anyway, even after a restart. It didn''t
make any
>> >difference to the FirePro in any case.
>> >
>> >I''d be grateful for some help.
>   I haven''t tried Windows 8, but on XP64 disabling the Cirrus card
in
>   device manager fixed the problem for me with Nvidia card passthrough.
>
>   You could try setting gfx_passthru=1, which should disable the
>   Cirrus emulation alltogether. That may make Windows domU completely
>   fail to boot, though.
>
>   Did you do a clean Win8 install and reboot the host fully? Is the
>   FirePro the primary GPU on the host? Or is it completely untouched
>   in dom0 (no dom0 or BIOS output on it)?
>
>   Gordan
Thanks for your reply. It turns out the problem was that I''d forgotten 
to put device_model_version="qemu-xen-traditional", so I was using a 
qemu that didn''t support gfx_passthrough. The FirePro is not used in 
dom0 at all. I''m using pciback to hide it.

I now have another problem though. The Ubuntu host crashes a lot when 
the Windows 8 guest is running. Mostly it happens when I try to shut 
down Windows - I think it might be a result of the FirePro card being 
released -, but sometimes it happens randomly too. The screen completely 
freezes, and no ssh or anything. I should mention that I''m not passing 
through the audio pci device for the FirePro, although it is bound to 
pciback so the host can''t be using it either. If I try to pass that 
through it never actually boots. I''ve had a lot of blue screens, and a 
lot of host crashes. I''m passing through a USB device too (i.e. the PCI
USB device), and that works fine.

Googling doesn''t seem to come up with much for host crashes. What can I
do to diagnose the problem? There''s nothing in the syslog.

Thanks a lot!

Toby

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

On 10/02/2013 03:01 PM, Toby Miller wrote:
>> On 2 Oct 2013, at 14:06, Gordan Bobic <gordan@bobich.net> wrote:
>>
>>> On 10/02/2013 01:58 PM, Toby Miller wrote:
>>>>> gordan@bobich.net wrote:
>>>>   On Sun, 29 Sep 2013 15:01:01 +0100, Toby
>>>> Miller<tobycmiller@gmail.com>   wrote:
>>>>>> I''m using Xen 4.3, installed from source on
Ubuntu 13.04. I''ve
>>>>>> successfully installed a Windows 8 guest, and had Xen
pass it an AMD
>>>>>> FirePro W7000 GPU (technically 2 PCI devices - the
second is for the
>>>>>> sound), as well as the virtualised graphics adapter.
Windows finds
>>>>>> the
>>>>>> card fine, and I''ve installed AMD''s
drivers. Device Manager says that
>>>>>> the card cannot find enough resources (code 12).
I''ve checked the IO,
>>>>>> IRQ, and Memory details in DM, and there don''t
seem to be any
>>>>>> clashes.
>>>>>>
>>>>>> I did try disabling the virtualised card from Windows,
but it seemed
>>>>>> to keep using it anyway, even after a restart. It
didn''t make any
>>>>>> difference to the FirePro in any case.
>>>>>>
>>>>>> I''d be grateful for some help.
>>>>   I haven''t tried Windows 8, but on XP64 disabling the
Cirrus card in
>>>>   device manager fixed the problem for me with Nvidia card
passthrough.
>>>>
>>>>   You could try setting gfx_passthru=1, which should disable
the
>>>>   Cirrus emulation alltogether. That may make Windows domU
completely
>>>>   fail to boot, though.
>>>>
>>>>   Did you do a clean Win8 install and reboot the host fully? Is
the
>>>>   FirePro the primary GPU on the host? Or is it completely
untouched
>>>>   in dom0 (no dom0 or BIOS output on it)?
>>>>
>>>>   Gordan
>>> Thanks for your reply. It turns out the problem was that
I''d forgotten
>>> to put device_model_version="qemu-xen-traditional", so I
was using a
>>> qemu that didn''t support gfx_passthrough. The FirePro is
not used in
>>> dom0 at all. I''m using pciback to hide it.
>>>
>>> I now have another problem though. The Ubuntu host crashes a lot
when
>>> the Windows 8 guest is running. Mostly it happens when I try to
shut
>>> down Windows - I think it might be a result of the FirePro card
being
>>> released -, but sometimes it happens randomly too. The screen
completely
>>> freezes, and no ssh or anything. I should mention that I''m
not passing
>>> through the audio pci device for the FirePro, although it is bound
to
>>> pciback so the host can''t be using it either. If I try to
pass that
>>> through it never actually boots. I''ve had a lot of blue
screens, and a
>>> lot of host crashes. I''m passing through a USB device too
(i.e. the PCI
>>> USB device), and that works fine.
>>>
>>> Googling doesn''t seem to come up with much for host
crashes. What can I
>>> do to diagnose the problem? There''s nothing in the syslog.
>>
>> Can you try it with domU limited to 1GB of RAM? Also, can you add your
physical e820 memory map as reported in dmesg?
>>
>> It is plausible you might be running into the same issue I had with
domU memory overwriting the physical IOMEM without getting remapped by the
IOMMU.
>>
>> Does your motherboard feature NF200 PCIe bridges?
>>
>> Gordan
>
> Your overwriting memory suggestion sounds likely. I''ve tried with
1GiB
> but the host still crashes a lot. The first time I ran it it booted,
> and then shutdown fine, but starting it again crashed the host.
That sounds like normal behaviour. Rebooting domU with ATI cards in 
passthrough does tend to result in crashing the host. I had this 
problem, too, without the memory overwriting issue.

This is one of the main reasons why I gave up on using ATI cards and 
switched to Nvidia. I don''t consider ATI cards fit for the purpose of 
VGA passthrough.
> The
> second and third tries resulted in a host crash before Windows had
> booted even once. Sometimes as the host crashes the mouse movement
> slows down, before stopping altogether. Would this make sense if the
> host''s memory were being overwritten?
No, that sounds like "normal" broken behaviour with ATI cards.
> Is this the memory map you wanted?:
>
> [root@yeti /etc/xen]% dmesg | grep e820
> [    0.000000] e820: BIOS-provided physical RAM map:
> [    0.000000] e820: update [mem 0x00000000-0x0000ffff] usable ==>
reserved
> [    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
> [    0.000000] e820: last_pfn = 0x880000 max_arch_pfn = 0x400000000
> [    0.000000] e820: last_pfn = 0x7d800 max_arch_pfn = 0x400000000
> [    0.000000] e820: [mem 0x90000000-0xfed1bfff] available for PCI devices
> [    8.827099] e820: reserve RAM buffer [mem 0x0009e000-0x0009ffff]
> [    8.827101] e820: reserve RAM buffer [mem 0x7cdad000-0x7fffffff]
> [    8.827103] e820: reserve RAM buffer [mem 0x7d334000-0x7fffffff]
> [    8.827105] e820: reserve RAM buffer [mem 0x7d800000-0x7fffffff]
Not quite, the block you are looking for is:
e820: BIOS-provided physical RAM map:
Xen: [mem 0x0000000000000000-0x000000000009cfff] usable
Xen: [mem 0x000000000009d000-0x00000000000fffff] reserved
Xen: [mem 0x0000000000100000-0x000000003f78ffff] usable
Xen: [mem 0x000000003f790000-0x000000003f79dfff] ACPI data
Xen: [mem 0x000000003f79e000-0x000000003f7cffff] ACPI NVS
Xen: [mem 0x000000003f7d0000-0x000000003f7dffff] reserved
Xen: [mem 0x000000003f7e7000-0x000000003fffffff] reserved
Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
Xen: [mem 0x00000000ffc00000-0x00000000ffffffff] reserved
Xen: [mem 0x0000000100000000-0x0000000cbfffffff] usable

> The motherboard is an EVGA SR-X, and I''m not sure how to tell
whether
> it has NF200 PCIe bridges.
I fear you''ll find it does. I have an SR-2. The SR-X almost certainly 
also has the same memory overwrite issue (NF200 bridges bypass the 
IOMMU) that I''ve been fighting. My advice, unless you are very bloody 
minded are prepared to either help write patches or use highly 
unsupported patches on self-built Xen packages, is to trade in your EVGA 
board for one made by a competent manufacturer. Otherwise you are in for 
a whole world of frustration and time wasting. I speak from experience. 
I''m persevering with the SR-2 because it has some tuning potential to 
it, unlike the SR-X, which makes the SR-X very much not worth bothering 
with. Persevere only if you deem your time worthless.

On the off-chance you do decide to persevere, despite the well 
intentioned advice above, you may want to look through the archives for 
a thread about HVM e820_host support. The patches on that thread are 
very much unfit for public consumption, but they do get around the 
memory stomp problem for me.

Unfortunately, I don''t think the symptoms you are seeing (domU crashing
on reboot) are to do with the memory stomp. It is a common Xen issue, 
somewhat less bad with certain quadrified Nvidia cards than with ATI 
cards. For example, I have 2 domUs with GPU passthrough. They are based 
off the same cloned image. One always handled rebooting OK (GTX470 
modified to Quadro 5000), one often seemed to fail to initialize the GPU 
(GTX480 modified to Quadro 6000). It _may_ have been related to what 
slot which card was in, I never quite got to the bottom of it. When it 
fails to init the Nvidia card, it would come up with emulated VGA output 
on VNC. I since upgraded the faux-Quadro 6000 to a faux-Quadro K5000 
(modified GTX680) and the problem went away - I can now reboot both 
domUs, but the GTX680 in domU has other issues (e.g. no output when 
using a dual-link DVI mode - something seemingly unique to my setup 
since David doesn''t have the same issue with his modified GTX680).

I have never managed to get an ATI passthrough setup to usably survive a 
domU reboot.

Gordan

On 2 Oct 2013, at 15:04, David TECHER <davidtecher@yahoo.fr> wrote:

AFAIR with Xen 4.3.0, Xen has to be patched with

ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch

cd /path/to/your/xen-4.3.0/directory

wget ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch -O
- | patch -p1

Install xen (make & Co).

This patch below was sent to Xen mailing list a couple months ago. It
should prevent you from getting error code 12.

Download the following script to manage domU.

ftp://ftp.enjellic.com/pub/xen/run-passthrough

Update it to your own needs.

When installing AMD driver avoid to install Catalyst Center -- to avoid
blue screen.

I did it several time with my HD 7970 and it worked perfectly.

http://www.davidgis.fr/blog/index.php?2013/07/13/954-xen-430-stable-vga-passthrourh-for-ati
http://www.davidgis.fr/blog/index.php?2013/04/05/937-xen-43-unstable-vga-passthrough-hd-7970-windows-7-64-bits-with-more-than-3gb-for-ram


Is it possible that this patch was already included in the Xen source I
downloaded about a week ago? It turns out that I was only getting code 12
when using the wrong qemu, so I guess it was being passed using standard
pci passthrough.

Catalyst Center is installed now, and the problems I''m having are
mostly
not blue-screen-related. I''ll remember that next time I''m
installing it
though.

Thanks a lot,

Toby



  ------------------------------
 *De :* Toby Miller <tobycmiller@gmail.com>
*À :* xen-users@lists.xen.org; gordan@bobich.net
*Envoyé le :* Mercredi 2 octobre 2013 14h58
*Objet :* Re: [Xen-users] VGA Passthrough of AMD FirePro W7000 to Windows
8: Not enough resources
>> gordan@bobich.net wrote:
>  On Sun, 29 Sep 2013 15:01:01 +0100, Toby
Miller<tobycmiller@gmail.com>
>  wrote:
>> >I''m using Xen 4.3, installed from source on Ubuntu 13.04.
I''ve
>> >successfully installed a Windows 8 guest, and had Xen pass it an
AMD
>> >FirePro W7000 GPU (technically 2 PCI devices - the second is for
the
>> >sound), as well as the virtualised graphics adapter. Windows finds
>> >the
>> >card fine, and I''ve installed AMD''s drivers.
Device Manager says that
>> >the card cannot find enough resources (code 12). I''ve
checked the IO,
>> >IRQ, and Memory details in DM, and there don''t seem to be
any
>> >clashes.
>> >
>> >I did try disabling the virtualised card from Windows, but it
seemed
>> >to keep using it anyway, even after a restart. It didn''t
make any
>> >difference to the FirePro in any case.
>> >
>> >I''d be grateful for some help.
>  I haven''t tried Windows 8, but on XP64 disabling the Cirrus card
in
>  device manager fixed the problem for me with Nvidia card passthrough.
>
>  You could try setting gfx_passthru=1, which should disable the
>  Cirrus emulation alltogether. That may make Windows domU completely
>  fail to boot, though.
>
>  Did you do a clean Win8 install and reboot the host fully? Is the
>  FirePro the primary GPU on the host? Or is it completely untouched
>  in dom0 (no dom0 or BIOS output on it)?
>
>  Gordan
Thanks for your reply. It turns out the problem was that I''d forgotten
to put device_model_version="qemu-xen-traditional", so I was using a
qemu that didn''t support gfx_passthrough. The FirePro is not used in
dom0 at all. I''m using pciback to hide it.

I now have another problem though. The Ubuntu host crashes a lot when
the Windows 8 guest is running. Mostly it happens when I try to shut
down Windows - I think it might be a result of the FirePro card being
released -, but sometimes it happens randomly too. The screen completely
freezes, and no ssh or anything. I should mention that I''m not passing
through the audio pci device for the FirePro, although it is bound to
pciback so the host can''t be using it either. If I try to pass that
through it never actually boots. I''ve had a lot of blue screens, and a
lot of host crashes. I''m passing through a USB device too (i.e. the PCI
USB device), and that works fine.

Googling doesn''t seem to come up with much for host crashes. What can I
do to diagnose the problem? There''s nothing in the syslog.

Thanks a lot!

Toby

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users


_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

On 02/10/13 15:17, Gordan Bobic wrote:
> On 10/02/2013 03:01 PM, Toby Miller wrote:
>>> On 2 Oct 2013, at 14:06, Gordan Bobic <gordan@bobich.net>
wrote:
>>>
>>>> On 10/02/2013 01:58 PM, Toby Miller wrote:
>>>>>> gordan@bobich.net wrote:
>>>>>   On Sun, 29 Sep 2013 15:01:01 +0100, Toby
>>>>> Miller<tobycmiller@gmail.com>   wrote:
>>>>>>> I''m using Xen 4.3, installed from source
on Ubuntu 13.04. I''ve
>>>>>>> successfully installed a Windows 8 guest, and had
Xen pass it an
>>>>>>> AMD
>>>>>>> FirePro W7000 GPU (technically 2 PCI devices - the
second is for
>>>>>>> the
>>>>>>> sound), as well as the virtualised graphics
adapter. Windows finds
>>>>>>> the
>>>>>>> card fine, and I''ve installed
AMD''s drivers. Device Manager says
>>>>>>> that
>>>>>>> the card cannot find enough resources (code 12).
I''ve checked
>>>>>>> the IO,
>>>>>>> IRQ, and Memory details in DM, and there
don''t seem to be any
>>>>>>> clashes.
>>>>>>>
>>>>>>> I did try disabling the virtualised card from
Windows, but it
>>>>>>> seemed
>>>>>>> to keep using it anyway, even after a restart. It
didn''t make any
>>>>>>> difference to the FirePro in any case.
>>>>>>>
>>>>>>> I''d be grateful for some help.
>>>>>   I haven''t tried Windows 8, but on XP64 disabling
the Cirrus card in
>>>>>   device manager fixed the problem for me with Nvidia card 
>>>>> passthrough.
>>>>>
>>>>>   You could try setting gfx_passthru=1, which should
disable the
>>>>>   Cirrus emulation alltogether. That may make Windows domU
completely
>>>>>   fail to boot, though.
>>>>>
>>>>>   Did you do a clean Win8 install and reboot the host
fully? Is the
>>>>>   FirePro the primary GPU on the host? Or is it completely
untouched
>>>>>   in dom0 (no dom0 or BIOS output on it)?
>>>>>
>>>>>   Gordan
>>>> Thanks for your reply. It turns out the problem was that
I''d forgotten
>>>> to put device_model_version="qemu-xen-traditional",
so I was using a
>>>> qemu that didn''t support gfx_passthrough. The FirePro
is not used in
>>>> dom0 at all. I''m using pciback to hide it.
>>>>
>>>> I now have another problem though. The Ubuntu host crashes a
lot when
>>>> the Windows 8 guest is running. Mostly it happens when I try to
shut
>>>> down Windows - I think it might be a result of the FirePro card
being
>>>> released -, but sometimes it happens randomly too. The screen 
>>>> completely
>>>> freezes, and no ssh or anything. I should mention that
I''m not passing
>>>> through the audio pci device for the FirePro, although it is
bound to
>>>> pciback so the host can''t be using it either. If I try
to pass that
>>>> through it never actually boots. I''ve had a lot of
blue screens, and a
>>>> lot of host crashes. I''m passing through a USB device
too (i.e. the
>>>> PCI
>>>> USB device), and that works fine.
>>>>
>>>> Googling doesn''t seem to come up with much for host
crashes. What
>>>> can I
>>>> do to diagnose the problem? There''s nothing in the
syslog.
>>>
>>> Can you try it with domU limited to 1GB of RAM? Also, can you add 
>>> your physical e820 memory map as reported in dmesg?
>>>
>>> It is plausible you might be running into the same issue I had with
>>> domU memory overwriting the physical IOMEM without getting remapped
>>> by the IOMMU.
>>>
>>> Does your motherboard feature NF200 PCIe bridges?
>>>
>>> Gordan
>>
>> Your overwriting memory suggestion sounds likely. I''ve tried
with 1GiB
>> but the host still crashes a lot. The first time I ran it it booted,
>> and then shutdown fine, but starting it again crashed the host.
>
> That sounds like normal behaviour. Rebooting domU with ATI cards in 
> passthrough does tend to result in crashing the host. I had this 
> problem, too, without the memory overwriting issue.
>
> This is one of the main reasons why I gave up on using ATI cards and 
> switched to Nvidia. I don''t consider ATI cards fit for the purpose
of
> VGA passthrough.
>
>> The
>> second and third tries resulted in a host crash before Windows had
>> booted even once. Sometimes as the host crashes the mouse movement
>> slows down, before stopping altogether. Would this make sense if the
>> host''s memory were being overwritten?
>
> No, that sounds like "normal" broken behaviour with ATI cards.
>
>> Is this the memory map you wanted?:
>>
>> [root@yeti /etc/xen]% dmesg | grep e820
>> [    0.000000] e820: BIOS-provided physical RAM map:
>> [    0.000000] e820: update [mem 0x00000000-0x0000ffff] usable ==> 
>> reserved
>> [    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
>> [    0.000000] e820: last_pfn = 0x880000 max_arch_pfn = 0x400000000
>> [    0.000000] e820: last_pfn = 0x7d800 max_arch_pfn = 0x400000000
>> [    0.000000] e820: [mem 0x90000000-0xfed1bfff] available for PCI 
>> devices
>> [    8.827099] e820: reserve RAM buffer [mem 0x0009e000-0x0009ffff]
>> [    8.827101] e820: reserve RAM buffer [mem 0x7cdad000-0x7fffffff]
>> [    8.827103] e820: reserve RAM buffer [mem 0x7d334000-0x7fffffff]
>> [    8.827105] e820: reserve RAM buffer [mem 0x7d800000-0x7fffffff]
>
> Not quite, the block you are looking for is:
> e820: BIOS-provided physical RAM map:
> Xen: [mem 0x0000000000000000-0x000000000009cfff] usable
> Xen: [mem 0x000000000009d000-0x00000000000fffff] reserved
> Xen: [mem 0x0000000000100000-0x000000003f78ffff] usable
> Xen: [mem 0x000000003f790000-0x000000003f79dfff] ACPI data
> Xen: [mem 0x000000003f79e000-0x000000003f7cffff] ACPI NVS
> Xen: [mem 0x000000003f7d0000-0x000000003f7dffff] reserved
> Xen: [mem 0x000000003f7e7000-0x000000003fffffff] reserved
> Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
> Xen: [mem 0x00000000ffc00000-0x00000000ffffffff] reserved
> Xen: [mem 0x0000000100000000-0x0000000cbfffffff] usable
>
>
>> The motherboard is an EVGA SR-X, and I''m not sure how to tell
whether
>> it has NF200 PCIe bridges.
>
> I fear you''ll find it does. I have an SR-2. The SR-X almost
certainly
> also has the same memory overwrite issue (NF200 bridges bypass the 
> IOMMU) that I''ve been fighting. My advice, unless you are very
bloody
> minded are prepared to either help write patches or use highly 
> unsupported patches on self-built Xen packages, is to trade in your 
> EVGA board for one made by a competent manufacturer. Otherwise you are 
> in for a whole world of frustration and time wasting. I speak from 
> experience. I''m persevering with the SR-2 because it has some
tuning
> potential to it, unlike the SR-X, which makes the SR-X very much not 
> worth bothering with. Persevere only if you deem your time worthless.
>
> On the off-chance you do decide to persevere, despite the well 
> intentioned advice above, you may want to look through the archives 
> for a thread about HVM e820_host support. The patches on that thread 
> are very much unfit for public consumption, but they do get around the 
> memory stomp problem for me.
>
> Unfortunately, I don''t think the symptoms you are seeing (domU 
> crashing on reboot) are to do with the memory stomp. It is a common 
> Xen issue, somewhat less bad with certain quadrified Nvidia cards than 
> with ATI cards. For example, I have 2 domUs with GPU passthrough. They 
> are based off the same cloned image. One always handled rebooting OK 
> (GTX470 modified to Quadro 5000), one often seemed to fail to 
> initialize the GPU (GTX480 modified to Quadro 6000). It _may_ have 
> been related to what slot which card was in, I never quite got to the 
> bottom of it. When it fails to init the Nvidia card, it would come up 
> with emulated VGA output on VNC. I since upgraded the faux-Quadro 6000 
> to a faux-Quadro K5000 (modified GTX680) and the problem went away - I 
> can now reboot both domUs, but the GTX680 in domU has other issues 
> (e.g. no output when using a dual-link DVI mode - something seemingly 
> unique to my setup since David doesn''t have the same issue with
his
> modified GTX680).
>
> I have never managed to get an ATI passthrough setup to usably survive 
> a domU reboot.
>
> Gordan
Thanks so much for all the advice. It seems to be stable enough once 
booted, so I think I''ll just accept that a reboot of Windows requires a
reboot of dom0 too. I don''t really need to be able to turn it off!

Best,

Toby

On Wed, 02 Oct 2013 18:12:11 +0100, Toby Miller <tobycmiller@gmail.com> 
 wrote:
> On 02/10/13 15:17, Gordan Bobic wrote:
>> On 10/02/2013 03:01 PM, Toby Miller wrote:
>>>> On 2 Oct 2013, at 14:06, Gordan Bobic <gordan@bobich.net>
wrote:
>>>>
>>>>> On 10/02/2013 01:58 PM, Toby Miller wrote:
>>>>>>> gordan@bobich.net wrote:
>>>>>>   On Sun, 29 Sep 2013 15:01:01 +0100, Toby
>>>>>> Miller<tobycmiller@gmail.com>   wrote:
>>>>>>>> I''m using Xen 4.3, installed from
source on Ubuntu 13.04. I''ve
>>>>>>>> successfully installed a Windows 8 guest, and
had Xen pass it
>>>>>>>> an AMD
>>>>>>>> FirePro W7000 GPU (technically 2 PCI devices -
the second is
>>>>>>>> for the
>>>>>>>> sound), as well as the virtualised graphics
adapter. Windows
>>>>>>>> finds
>>>>>>>> the
>>>>>>>> card fine, and I''ve installed
AMD''s drivers. Device Manager
>>>>>>>> says that
>>>>>>>> the card cannot find enough resources (code
12). I''ve checked
>>>>>>>> the IO,
>>>>>>>> IRQ, and Memory details in DM, and there
don''t seem to be any
>>>>>>>> clashes.
>>>>>>>>
>>>>>>>> I did try disabling the virtualised card from
Windows, but it
>>>>>>>> seemed
>>>>>>>> to keep using it anyway, even after a restart.
It didn''t make
>>>>>>>> any
>>>>>>>> difference to the FirePro in any case.
>>>>>>>>
>>>>>>>> I''d be grateful for some help.
>>>>>>   I haven''t tried Windows 8, but on XP64
disabling the Cirrus
>>>>>> card in
>>>>>>   device manager fixed the problem for me with Nvidia
card
>>>>>> passthrough.
>>>>>>
>>>>>>   You could try setting gfx_passthru=1, which should
disable the
>>>>>>   Cirrus emulation alltogether. That may make Windows
domU
>>>>>> completely
>>>>>>   fail to boot, though.
>>>>>>
>>>>>>   Did you do a clean Win8 install and reboot the host
fully? Is
>>>>>> the
>>>>>>   FirePro the primary GPU on the host? Or is it
completely
>>>>>> untouched
>>>>>>   in dom0 (no dom0 or BIOS output on it)?
>>>>>>
>>>>>>   Gordan
>>>>> Thanks for your reply. It turns out the problem was that
I''d
>>>>> forgotten
>>>>> to put
device_model_version="qemu-xen-traditional", so I was
>>>>> using a
>>>>> qemu that didn''t support gfx_passthrough. The
FirePro is not used
>>>>> in
>>>>> dom0 at all. I''m using pciback to hide it.
>>>>>
>>>>> I now have another problem though. The Ubuntu host crashes
a lot
>>>>> when
>>>>> the Windows 8 guest is running. Mostly it happens when I
try to
>>>>> shut
>>>>> down Windows - I think it might be a result of the FirePro
card
>>>>> being
>>>>> released -, but sometimes it happens randomly too. The
screen
>>>>> completely
>>>>> freezes, and no ssh or anything. I should mention that
I''m not
>>>>> passing
>>>>> through the audio pci device for the FirePro, although it
is
>>>>> bound to
>>>>> pciback so the host can''t be using it either. If I
try to pass
>>>>> that
>>>>> through it never actually boots. I''ve had a lot of
blue screens,
>>>>> and a
>>>>> lot of host crashes. I''m passing through a USB
device too (i.e.
>>>>> the PCI
>>>>> USB device), and that works fine.
>>>>>
>>>>> Googling doesn''t seem to come up with much for
host crashes. What
>>>>> can I
>>>>> do to diagnose the problem? There''s nothing in the
syslog.
>>>>
>>>> Can you try it with domU limited to 1GB of RAM? Also, can you
add
>>>> your physical e820 memory map as reported in dmesg?
>>>>
>>>> It is plausible you might be running into the same issue I had 
>>>> with domU memory overwriting the physical IOMEM without getting
>>>> remapped by the IOMMU.
>>>>
>>>> Does your motherboard feature NF200 PCIe bridges?
>>>>
>>>> Gordan
>>>
>>> Your overwriting memory suggestion sounds likely. I''ve
tried with
>>> 1GiB
>>> but the host still crashes a lot. The first time I ran it it 
>>> booted,
>>> and then shutdown fine, but starting it again crashed the host.
>>
>> That sounds like normal behaviour. Rebooting domU with ATI cards in 
>> passthrough does tend to result in crashing the host. I had this 
>> problem, too, without the memory overwriting issue.
>>
>> This is one of the main reasons why I gave up on using ATI cards and 
>> switched to Nvidia. I don''t consider ATI cards fit for the
purpose of
>> VGA passthrough.
>>
>>> The
>>> second and third tries resulted in a host crash before Windows had
>>> booted even once. Sometimes as the host crashes the mouse movement
>>> slows down, before stopping altogether. Would this make sense if 
>>> the
>>> host''s memory were being overwritten?
>>
>> No, that sounds like "normal" broken behaviour with ATI
cards.
>>
>>> Is this the memory map you wanted?:
>>>
>>> [root@yeti /etc/xen]% dmesg | grep e820
>>> [    0.000000] e820: BIOS-provided physical RAM map:
>>> [    0.000000] e820: update [mem 0x00000000-0x0000ffff] usable
==>
>>> reserved
>>> [    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
>>> [    0.000000] e820: last_pfn = 0x880000 max_arch_pfn = 0x400000000
>>> [    0.000000] e820: last_pfn = 0x7d800 max_arch_pfn = 0x400000000
>>> [    0.000000] e820: [mem 0x90000000-0xfed1bfff] available for PCI 
>>> devices
>>> [    8.827099] e820: reserve RAM buffer [mem 0x0009e000-0x0009ffff]
>>> [    8.827101] e820: reserve RAM buffer [mem 0x7cdad000-0x7fffffff]
>>> [    8.827103] e820: reserve RAM buffer [mem 0x7d334000-0x7fffffff]
>>> [    8.827105] e820: reserve RAM buffer [mem 0x7d800000-0x7fffffff]
>>
>> Not quite, the block you are looking for is:
>> e820: BIOS-provided physical RAM map:
>> Xen: [mem 0x0000000000000000-0x000000000009cfff] usable
>> Xen: [mem 0x000000000009d000-0x00000000000fffff] reserved
>> Xen: [mem 0x0000000000100000-0x000000003f78ffff] usable
>> Xen: [mem 0x000000003f790000-0x000000003f79dfff] ACPI data
>> Xen: [mem 0x000000003f79e000-0x000000003f7cffff] ACPI NVS
>> Xen: [mem 0x000000003f7d0000-0x000000003f7dffff] reserved
>> Xen: [mem 0x000000003f7e7000-0x000000003fffffff] reserved
>> Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
>> Xen: [mem 0x00000000ffc00000-0x00000000ffffffff] reserved
>> Xen: [mem 0x0000000100000000-0x0000000cbfffffff] usable
>>
>>
>>> The motherboard is an EVGA SR-X, and I''m not sure how to
tell
>>> whether
>>> it has NF200 PCIe bridges.
>>
>> I fear you''ll find it does. I have an SR-2. The SR-X almost 
>> certainly also has the same memory overwrite issue (NF200 bridges 
>> bypass the IOMMU) that I''ve been fighting. My advice, unless
you are
>> very bloody minded are prepared to either help write patches or use 
>> highly unsupported patches on self-built Xen packages, is to trade in 
>> your EVGA board for one made by a competent manufacturer. Otherwise 
>> you are in for a whole world of frustration and time wasting. I speak 
>> from experience. I''m persevering with the SR-2 because it has
some
>> tuning potential to it, unlike the SR-X, which makes the SR-X very 
>> much not worth bothering with. Persevere only if you deem your time 
>> worthless.
>>
>> On the off-chance you do decide to persevere, despite the well 
>> intentioned advice above, you may want to look through the archives 
>> for a thread about HVM e820_host support. The patches on that thread 
>> are very much unfit for public consumption, but they do get around the 
>> memory stomp problem for me.
>>
>> Unfortunately, I don''t think the symptoms you are seeing (domU
>> crashing on reboot) are to do with the memory stomp. It is a common 
>> Xen issue, somewhat less bad with certain quadrified Nvidia cards than 
>> with ATI cards. For example, I have 2 domUs with GPU passthrough. They 
>> are based off the same cloned image. One always handled rebooting OK 
>> (GTX470 modified to Quadro 5000), one often seemed to fail to 
>> initialize the GPU (GTX480 modified to Quadro 6000). It _may_ have 
>> been related to what slot which card was in, I never quite got to the 
>> bottom of it. When it fails to init the Nvidia card, it would come up 
>> with emulated VGA output on VNC. I since upgraded the faux-Quadro 6000 
>> to a faux-Quadro K5000 (modified GTX680) and the problem went away - I 
>> can now reboot both domUs, but the GTX680 in domU has other issues 
>> (e.g. no output when using a dual-link DVI mode - something seemingly 
>> unique to my setup since David doesn''t have the same issue
with his
>> modified GTX680).
>>
>> I have never managed to get an ATI passthrough setup to usably 
>> survive a domU reboot.
>>
>> Gordan
>
> Thanks so much for all the advice. It seems to be stable enough once
> booted, so I think I''ll just accept that a reboot of Windows
requires
> a reboot of dom0 too. I don''t really need to be able to turn it
off!
 One thing that some people have found helps on Windows 7 and later
 is to eject the GPU as a hot-plug device (like you would dismount
 a USB stick, from the removable hardware icon in the tray) before
 rebooting. The driver seems to know how to reset the card into a
 clean state if you do that. The only problem is that you then lose
 GPU output which makes shutting down difficult.

 What you might be able to do is write something in powershell that
 hooks on shutdown and ejects the GPU.

 I never got as far as working around that level of issues with ATI
 cards because their limitations proved to be completely unworkable
 with my T221 monitors.

 Gordan

On 04/10/13 14:36, Gordan Bobic wrote:
> On Wed, 02 Oct 2013 18:12:11 +0100, Toby Miller 
> <tobycmiller@gmail.com> wrote:
>> On 02/10/13 15:17, Gordan Bobic wrote:
>>> On 10/02/2013 03:01 PM, Toby Miller wrote:
>>>>> On 2 Oct 2013, at 14:06, Gordan Bobic
<gordan@bobich.net> wrote:
>>>>>
>>>>>> On 10/02/2013 01:58 PM, Toby Miller wrote:
>>>>>>>> gordan@bobich.net wrote:
>>>>>>>   On Sun, 29 Sep 2013 15:01:01 +0100, Toby
>>>>>>> Miller<tobycmiller@gmail.com>   wrote:
>>>>>>>>> I''m using Xen 4.3, installed from
source on Ubuntu 13.04. I''ve
>>>>>>>>> successfully installed a Windows 8 guest,
and had Xen pass it
>>>>>>>>> an AMD
>>>>>>>>> FirePro W7000 GPU (technically 2 PCI
devices - the second is
>>>>>>>>> for the
>>>>>>>>> sound), as well as the virtualised graphics
adapter. Windows
>>>>>>>>> finds
>>>>>>>>> the
>>>>>>>>> card fine, and I''ve installed
AMD''s drivers. Device Manager
>>>>>>>>> says that
>>>>>>>>> the card cannot find enough resources (code
12). I''ve checked
>>>>>>>>> the IO,
>>>>>>>>> IRQ, and Memory details in DM, and there
don''t seem to be any
>>>>>>>>> clashes.
>>>>>>>>>
>>>>>>>>> I did try disabling the virtualised card
from Windows, but it
>>>>>>>>> seemed
>>>>>>>>> to keep using it anyway, even after a
restart. It didn''t make any
>>>>>>>>> difference to the FirePro in any case.
>>>>>>>>>
>>>>>>>>> I''d be grateful for some help.
>>>>>>>   I haven''t tried Windows 8, but on XP64
disabling the Cirrus
>>>>>>> card in
>>>>>>>   device manager fixed the problem for me with
Nvidia card
>>>>>>> passthrough.
>>>>>>>
>>>>>>>   You could try setting gfx_passthru=1, which
should disable the
>>>>>>>   Cirrus emulation alltogether. That may make
Windows domU
>>>>>>> completely
>>>>>>>   fail to boot, though.
>>>>>>>
>>>>>>>   Did you do a clean Win8 install and reboot the
host fully? Is the
>>>>>>>   FirePro the primary GPU on the host? Or is it
completely
>>>>>>> untouched
>>>>>>>   in dom0 (no dom0 or BIOS output on it)?
>>>>>>>
>>>>>>>   Gordan
>>>>>> Thanks for your reply. It turns out the problem was
that I''d
>>>>>> forgotten
>>>>>> to put
device_model_version="qemu-xen-traditional", so I was using a
>>>>>> qemu that didn''t support gfx_passthrough. The
FirePro is not used in
>>>>>> dom0 at all. I''m using pciback to hide it.
>>>>>>
>>>>>> I now have another problem though. The Ubuntu host
crashes a lot
>>>>>> when
>>>>>> the Windows 8 guest is running. Mostly it happens when
I try to shut
>>>>>> down Windows - I think it might be a result of the
FirePro card
>>>>>> being
>>>>>> released -, but sometimes it happens randomly too. The
screen
>>>>>> completely
>>>>>> freezes, and no ssh or anything. I should mention that
I''m not
>>>>>> passing
>>>>>> through the audio pci device for the FirePro, although
it is
>>>>>> bound to
>>>>>> pciback so the host can''t be using it either.
If I try to pass that
>>>>>> through it never actually boots. I''ve had a
lot of blue screens,
>>>>>> and a
>>>>>> lot of host crashes. I''m passing through a USB
device too (i.e.
>>>>>> the PCI
>>>>>> USB device), and that works fine.
>>>>>>
>>>>>> Googling doesn''t seem to come up with much for
host crashes. What
>>>>>> can I
>>>>>> do to diagnose the problem? There''s nothing in
the syslog.
>>>>>
>>>>> Can you try it with domU limited to 1GB of RAM? Also, can
you add
>>>>> your physical e820 memory map as reported in dmesg?
>>>>>
>>>>> It is plausible you might be running into the same issue I
had
>>>>> with domU memory overwriting the physical IOMEM without
getting
>>>>> remapped by the IOMMU.
>>>>>
>>>>> Does your motherboard feature NF200 PCIe bridges?
>>>>>
>>>>> Gordan
>>>>
>>>> Your overwriting memory suggestion sounds likely. I''ve
tried with 1GiB
>>>> but the host still crashes a lot. The first time I ran it it
booted,
>>>> and then shutdown fine, but starting it again crashed the host.
>>>
>>> That sounds like normal behaviour. Rebooting domU with ATI cards in
>>> passthrough does tend to result in crashing the host. I had this 
>>> problem, too, without the memory overwriting issue.
>>>
>>> This is one of the main reasons why I gave up on using ATI cards
and
>>> switched to Nvidia. I don''t consider ATI cards fit for the
purpose
>>> of VGA passthrough.
>>>
>>>> The
>>>> second and third tries resulted in a host crash before Windows
had
>>>> booted even once. Sometimes as the host crashes the mouse
movement
>>>> slows down, before stopping altogether. Would this make sense
if the
>>>> host''s memory were being overwritten?
>>>
>>> No, that sounds like "normal" broken behaviour with ATI
cards.
>>>
>>>> Is this the memory map you wanted?:
>>>>
>>>> [root@yeti /etc/xen]% dmesg | grep e820
>>>> [    0.000000] e820: BIOS-provided physical RAM map:
>>>> [    0.000000] e820: update [mem 0x00000000-0x0000ffff] usable
==>
>>>> reserved
>>>> [    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
>>>> [    0.000000] e820: last_pfn = 0x880000 max_arch_pfn =
0x400000000
>>>> [    0.000000] e820: last_pfn = 0x7d800 max_arch_pfn =
0x400000000
>>>> [    0.000000] e820: [mem 0x90000000-0xfed1bfff] available for
PCI
>>>> devices
>>>> [    8.827099] e820: reserve RAM buffer [mem
0x0009e000-0x0009ffff]
>>>> [    8.827101] e820: reserve RAM buffer [mem
0x7cdad000-0x7fffffff]
>>>> [    8.827103] e820: reserve RAM buffer [mem
0x7d334000-0x7fffffff]
>>>> [    8.827105] e820: reserve RAM buffer [mem
0x7d800000-0x7fffffff]
>>>
>>> Not quite, the block you are looking for is:
>>> e820: BIOS-provided physical RAM map:
>>> Xen: [mem 0x0000000000000000-0x000000000009cfff] usable
>>> Xen: [mem 0x000000000009d000-0x00000000000fffff] reserved
>>> Xen: [mem 0x0000000000100000-0x000000003f78ffff] usable
>>> Xen: [mem 0x000000003f790000-0x000000003f79dfff] ACPI data
>>> Xen: [mem 0x000000003f79e000-0x000000003f7cffff] ACPI NVS
>>> Xen: [mem 0x000000003f7d0000-0x000000003f7dffff] reserved
>>> Xen: [mem 0x000000003f7e7000-0x000000003fffffff] reserved
>>> Xen: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
>>> Xen: [mem 0x00000000ffc00000-0x00000000ffffffff] reserved
>>> Xen: [mem 0x0000000100000000-0x0000000cbfffffff] usable
>>>
>>>
>>>> The motherboard is an EVGA SR-X, and I''m not sure how
to tell whether
>>>> it has NF200 PCIe bridges.
>>>
>>> I fear you''ll find it does. I have an SR-2. The SR-X
almost
>>> certainly also has the same memory overwrite issue (NF200 bridges 
>>> bypass the IOMMU) that I''ve been fighting. My advice,
unless you are
>>> very bloody minded are prepared to either help write patches or use
>>> highly unsupported patches on self-built Xen packages, is to trade 
>>> in your EVGA board for one made by a competent manufacturer. 
>>> Otherwise you are in for a whole world of frustration and time 
>>> wasting. I speak from experience. I''m persevering with the
SR-2
>>> because it has some tuning potential to it, unlike the SR-X, which 
>>> makes the SR-X very much not worth bothering with. Persevere only
if
>>> you deem your time worthless.
>>>
>>> On the off-chance you do decide to persevere, despite the well 
>>> intentioned advice above, you may want to look through the archives
>>> for a thread about HVM e820_host support. The patches on that
thread
>>> are very much unfit for public consumption, but they do get around 
>>> the memory stomp problem for me.
>>>
>>> Unfortunately, I don''t think the symptoms you are seeing
(domU
>>> crashing on reboot) are to do with the memory stomp. It is a common
>>> Xen issue, somewhat less bad with certain quadrified Nvidia cards 
>>> than with ATI cards. For example, I have 2 domUs with GPU 
>>> passthrough. They are based off the same cloned image. One always 
>>> handled rebooting OK (GTX470 modified to Quadro 5000), one often 
>>> seemed to fail to initialize the GPU (GTX480 modified to Quadro 
>>> 6000). It _may_ have been related to what slot which card was in, I
>>> never quite got to the bottom of it. When it fails to init the 
>>> Nvidia card, it would come up with emulated VGA output on VNC. I 
>>> since upgraded the faux-Quadro 6000 to a faux-Quadro K5000
(modified
>>> GTX680) and the problem went away - I can now reboot both domUs,
but
>>> the GTX680 in domU has other issues (e.g. no output when using a 
>>> dual-link DVI mode - something seemingly unique to my setup since 
>>> David doesn''t have the same issue with his modified
GTX680).
>>>
>>> I have never managed to get an ATI passthrough setup to usably 
>>> survive a domU reboot.
>>>
>>> Gordan
>>
>> Thanks so much for all the advice. It seems to be stable enough once
>> booted, so I think I''ll just accept that a reboot of Windows
requires
>> a reboot of dom0 too. I don''t really need to be able to turn
it off!
>
> One thing that some people have found helps on Windows 7 and later
> is to eject the GPU as a hot-plug device (like you would dismount
> a USB stick, from the removable hardware icon in the tray) before
> rebooting. The driver seems to know how to reset the card into a
> clean state if you do that. The only problem is that you then lose
> GPU output which makes shutting down difficult.
>
> What you might be able to do is write something in powershell that
> hooks on shutdown and ejects the GPU.
>
> I never got as far as working around that level of issues with ATI
> cards because their limitations proved to be completely unworkable
> with my T221 monitors.
>
> Gordan
That''s a good idea. I''ll try that.

Thanks,

Toby

On Oct 2, 2013, at 10:19 AM, Toby Miller <tobycmiller@gmail.com> wrote:
> 
> On 2 Oct 2013, at 15:04, David TECHER <davidtecher@yahoo.fr> wrote:
> 
>> AFAIR with Xen 4.3.0, Xen has to be patched with 
>> ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch
>> cd /path/to/your/xen-4.3.0/directory
>> wget ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch -O
- | patch -p1
>> Install xen (make & Co).
>> 
>> This patch below was sent to Xen mailing list a couple months ago. It
should prevent you from getting error code 12.
>> 
>> Download the following script to manage domU.
>> 
>> ftp://ftp.enjellic.com/pub/xen/run-passthrough
>> 
>> Update it to your own needs.
>> 
>> When installing AMD driver avoid to install Catalyst Center -- to avoid
blue screen.
>> 
>> I did it several time with my HD 7970 and it worked perfectly.
>> 
>>
http://www.davidgis.fr/blog/index.php?2013/07/13/954-xen-430-stable-vga-passthrourh-for-ati
>>
http://www.davidgis.fr/blog/index.php?2013/04/05/937-xen-43-unstable-vga-passthrough-hd-7970-windows-7-64-bits-with-more-than-3gb-for-ram
> 
> Is it possible that this patch was already included in the Xen source I
downloaded about a week ago? It turns out that I was only getting code 12 when
using the wrong qemu, so I guess it was being passed using standard pci
passthrough.
> 
> Catalyst Center is installed now, and the problems I''m having are
mostly not blue-screen-related. I''ll remember that next time
I''m installing it though.
Hello Toby,

Glad it''s working!

I recall reading that the problem where a CCC install would bork the VM with a
100% chance was solved somewhere between the 12.x to 13.0 Radeon drivers. 
Still, I''ve seen *some* VMs behave well and others fail spectacularly. 
I never got it to the point where I was satisfied what the cause is, but the
general rule I follow now is:

1) install bare driver, reboot host.

2) create a restore point (or snapshot the VM disk), then run the install again
and install CCC (or VISION or whatever the name is now :P)

Just a note for reference and any stray, future googlers!

Cheers,
Andrew

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users