Xen users - Sep 2013 - Xen VGA Passthrough - GTX 480 successfully quadrified to quadro 6000 (softmod) - more than 4GB of RAM for Win XP 64 Bits

Hi Gordan,

I received my Asus GTX 480 today (bought on ebay).

Finally I succeeded in 

- quadrifying my GTX 480  into a Quadro 6000 ( softmod). 

- overpassing the 3-4GB of RAM on Windows XP 64 bits (domU) by applying a well
known patch for Xen.

- playing Crysis 2 and Darksiders II for a few minutes.

http://img11.hostingpics.net/pics/953254ScreenshotXenSoftModedQuadro6000.png

Using nvlfash I had to alter my (OR mask 1) so I could install the latest driver
for Quadro 6000
(320.86-quadro-tesla-grid-desktop-winxp-64bit-international-whql.exe).

I noticed that I had to do that else my domU may freez while launching Crysis 2.

Everything works perfectly for the moment.

I was very happy when my domU started without to apply my annoying patches for
NVIDIA :)

I will have a try for a Linux domU tomorrow (Xen PVonHVM + NVIDIA + Wine). This
is what I want :)

Kind regards.

David

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

On 09/17/2013 01:11 AM, David TECHER wrote:

Good to see somebody else has this working. :)
> Finally I succeeded in
>
> - quadrifying my GTX 480  into a Quadro 6000 ( softmod).
>
> - overpassing the 3-4GB of RAM on Windows XP 64 bits (domU) by applying
> a well known patch for Xen.
Can you elaborate? Are you referring to the 64-bit IOMEM remap or 
something else?
> Using nvlfash I had to alter my (OR mask 1) so I could install the
> latest driver for Quadro 6000
> (320.86-quadro-tesla-grid-desktop-winxp-64bit-international-whql.exe).
>
> I noticed that I had to do that else my domU may freez while launching
> Crysis 2.
You mean you needed the latest driver? That''s odd - I have used all the
drivers from 320.00 onward and they all worked forme. Or are you saying 
you could boot the VM with the card as a GTX480, but the game would lock 
up when starting?

I tried getting the GeForce cards to work with VGA passthrough but I 
couldn''t get it working without quadrifying them. After that everything
has "just worked" as expected.
> Everything works perfectly for the moment.
>
> I was very happy when my domU started without to apply my annoying
> patches for NVIDIA :)
I''m working on making some of those more dynamic for implementing 
e820_host for HVMs. Once that is completed it should also ensure 
vBAR=pBAR - with a bit of luck this will make GeForce cards work without 
quadrifying them.
> I will have a try for a Linux domU tomorrow (Xen PVonHVM + NVIDIA +
> Wine). This is what I want :)
Please, do report back how you get on - I don''t think anyone has tried 
that before. :)

Gordan

Gordan,


I am able to have more than 3-4GB of RAM without any patch. So no patch 
required! It works both for domU = Linux 64 and domU = Win7 64.So I was mistaken
here.


My domU on XP 64 was locked up (frozen) when game was starting. I had to change
my (OR mask 1) value for soft straps so this issue went away.
Whatever driver used it works like a charm.

I set up a new VM for Win7 64 with RAM=8GB (no patch for Xen!). I was able to
play ''Remember me'' :)

Here is a screenshot

http://img15.hostingpics.net/pics/654044ScreenshotXenSoftmodedQuadro6000Win764.png

I am working on testing games on Wine with (Xen PVONHVM+NVIDIA). 


I have great esteem  for the work you did/are currently doing. Won''t be
back to Xen/NVIDIA world without you :)



"GTX 500 series + soldering iron" are for professional guys like you
:). Got a GTX 680 at home.I will wait for your next article on your web site
before


Please keep us posted about your investigations for e820_host for HVMs


Kind regards.






________________________________
 De : Gordan Bobic <gordan@bobich.net>
À : David TECHER <davidtecher@yahoo.fr> 
Cc : "xen-users@lists.xen.org" <xen-users@lists.xen.org> 
Envoyé le : Mardi 17 septembre 2013 16h35
Objet : Re: [Xen-users] Xen VGA Passthrough - GTX 480 successfully quadrified to
quadro 6000 (softmod) - more than 4GB of RAM for Win XP 64 Bits

On 09/17/2013 01:11 AM, David TECHER wrote:

Good to see somebody else has this working. :)
> Finally I succeeded in
>
> - quadrifying my GTX 480  into a Quadro 6000 ( softmod).
>
> - overpassing the 3-4GB of RAM on Windows XP 64 bits (domU) by applying
> a well known patch for Xen.
Can you elaborate? Are you referring to the 64-bit IOMEM remap or 
something else?
> Using nvlfash I had to alter my (OR mask 1) so I could install the
> latest driver for Quadro 6000
> (320.86-quadro-tesla-grid-desktop-winxp-64bit-international-whql.exe).
>
> I noticed that I had to do that else my domU may freez while launching
> Crysis 2.
You mean you needed the latest driver? That''s odd - I have used all the
drivers from 320.00 onward and they all worked forme. Or are you saying 
you could boot the VM with the card as a GTX480, but the game would lock 
up when starting?

I tried getting the GeForce cards to work with VGA passthrough but I 
couldn''t get it working without quadrifying them. After that everything
has "just worked" as expected.
> Everything works perfectly for the moment.
>
> I was very happy when my domU started without to apply my annoying
> patches for NVIDIA :)
I''m working on making some of those more dynamic for implementing 
e820_host for HVMs. Once that is completed it should also ensure 
vBAR=pBAR - with a bit of luck this will make GeForce cards work without 
quadrifying them.
> I will have a try for a Linux domU tomorrow (Xen PVonHVM + NVIDIA +
> Wine). This is what I want :)
Please, do report back how you get on - I don''t think anyone has tried 
that before. :)

Gordan


_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

On 09/17/2013 09:27 PM, David TECHER wrote:
> Gordan,
>
> I am able to have more than 3-4GB of RAM without any patch. So no patch
> required! It works both for domU = Linux 64 and domU = Win7 64.
> So I was mistaken here.
Oh, I can believe you need a patch - the only reason I can pass more 
than 2688MB of RAM to my VMs is because I patched hvmloader to make sure 
that it leaves memory holes in the same places as the host - because my 
dodgy PCIe bridges bypass the IOMMU and when the host''s virtual memory 
addresses equal the physical device''s BAR, everything crashes
spectacularly.

But I''m glad you don''t have hardware as broken as me. :)
> My domU on XP 64 was locked up (frozen) when game was starting. I had to
> change my (OR mask 1) value for soft straps so this issue went away.
> Whatever driver used it works like a charm.
Hang on - you mean to say that you managed to get VGA output on an 
external monitor with VGA passthrough WITHOUT quadrifying the card? 
Before I quadrified mine I couldn''t get anything at all - VNC console 
only, and device manager was saying it couldn''t start the device.

It makes me wonder if Nvidia are now tweaking the GeForce drivers to 
work better to discourage people from looking at modifying cards from 
GeForce to Quadro (since VGA passthrough is _the_ main benefit). Maybe 
they are concerned about what else we might figure out. ;)
> I set up a new VM for Win7 64 with RAM=8GB (no patch for Xen!). I was
> able to play ''Remember me'' :)
>
> Here is a screenshot
>
>
http://img15.hostingpics.net/pics/654044ScreenshotXenSoftmodedQuadro6000Win764.png
Good, good. Did you make sure you changed the PCI device ID in BIOS at 
around 0x18E ? This should be consistent with the PCI device ID.
> I am working on testing games on Wine with (Xen PVONHVM+NVIDIA).
>
>
> I have great esteem  for the work you did/are currently doing.
Won''t be
> back to Xen/NVIDIA world without you :)
I''m afraid I cannot take that much credit - to paraphrase the old
quote,
I have only succeeded at this by "standing on the shoulders of giants"
that reverse engineered bits of this before me.
> "GTX 500 series + soldering iron" are for professional guys like
you :).
Ironically, GTX580 is somewhat pointless to modify. You can soft-mod it 
to a Quadro 7000, but that isn''t MultiOS, so VGA passthrough
doesn''t
work (having said that - Quadro K5000 isn''t MultiOS either but VGA 
passthrough works great). There is no SPEC boost from modding the 580, 
and the 580 has no bidirectional DMA engine to enable. So the only 
narrow use-case for modding it is for some professional applications (I 
seem to recall some Adobe apps were mentioned ages ago for refusing to 
even acknowledge the presence of a GeForce card for GPU processing 
offload, but worked fine with Quadros, so in that instance there may be 
  benefit (then again, I seem to recall the same could be achieved with 
an .inf mod without modifying the BIOS at all - sadly that doesn''t seem
to work for VGA passthrough).
> Got a GTX 680 at home. I will wait for your next article on your web
> site before
OK. That can be soft-modded to a Tesla K10, but requires a hard-mod for 
K5000/K2 conversion. So you''ll have to get your soldering iron out for 
it, unfortunately. The process is well documented on the eevblog forum. 
What I can bring to the table is BIOS modding to get a full K5000 BIOS 
onto a 4GB GTX680 (will probably work just the same on a GTX770 since 
they are virtually identical). What I might work on next is stripping 
out all the UEFI wrappers so the real BIOS fits in under the 64KB limit 
for primary gfx passthrough. :)
> Please keep us posted about your investigations for e820_host for HVMs
You can keep track of it on the xen-devel list, but I haven''t had a 
chance to do anything over and above the proof of concept patches yet.

Gordan

I use mini-HDMI port on my card.

Yes I modified the  PCI device ID in BIOS at around 0x18E. It could be done
using a hex editor. Insteaf of using a hex editor you can use Nibitor.

Nibitor 6.06 has support for Quadro 6000 in its devices list. You can have a
look here

http://www.davidgis.fr/blog/index.php?2013/09/08/965-update-asus-gtx-480-bios-to-quadro-fx-6000-bios

Moreover Nibitor 6.06 could be used on Linux using Wine :).

To be more understandable the expected command is -- as you already know --.

nvflash --straps (and mask 0) (or mask 0) (and mask 1) (or mask 1)

Of course here you need to have a hex editor so you have the 4 hex values.

I just read instructions from
http://www.davidgis.fr/blog/index.php?2013/09/08/965-update-asus-gtx-480-bios-to-quadro-fx-6000-bios
(referring  to the 7th message).

It took me several days  to understand it. ( little endian? straps? softs
straps? soft mod?). Several researchs on Internet were required to understand it
when you are a noob as I am.

Only (or mask 0) is supposed to be updated.

I just get  the 4 hex values. After that

(and mask 0): original value

(or mask 0): updated as expected

(and mask 1): original value

(or mask 1):  updated so my issue on my domU ( XP 64)  went away -- when Crysis
2 was starting and was locking up --.

Don''t ask me why it fixed my issue.  I decided to modify the (or mask
1) knowing that it could break something for my card. Let''s say that I
was lucky :)

I just modified both (or mak 0) and (or mask 1). I just noticed that it works.
It is enough for me.

Playing Windows games on a Linux domU using Wine could be problematic with a
such card. This quadrified card is a GTX 480 -- well known to have temperature
problems --.

At home I have two GTX 480 cards.

1) #1 is used actually for my current tests

2) #2 with Arctic Cooling Accelero XTREME Plus. I will try to do the same tests
on this one. This one should fixed  temperature problems :)

------------------
For e820_host: I am reading your mails on xen-devel list so I am updated :)

For GTX 680/GTX 770:  it is clear that the 64KB limit is the first problem --
primary gfx passthrough --. If you succeeded then keep us posted.






________________________________
 De : Gordan Bobic <gordan@bobich.net>
À : David TECHER <davidtecher@yahoo.fr> 
Cc : "xen-users@lists.xen.org" <xen-users@lists.xen.org> 
Envoyé le : Mardi 17 septembre 2013 23h06
Objet : Re: [Xen-users] Xen VGA Passthrough - GTX 480 successfully quadrified to
quadro 6000 (softmod) - more than 4GB of RAM for Win XP 64 Bits
 

On 09/17/2013 09:27 PM, David TECHER wrote:
> Gordan,
>
> I am able to have more than 3-4GB of RAM without any patch. So no patch
> required! It works both for domU = Linux 64 and domU = Win7 64.
> So I was mistaken here.
Oh, I can believe you need a patch - the only reason I can pass more 
than 2688MB of RAM to my VMs is because I patched hvmloader to make sure 
that it leaves memory holes in the same places as the host - because my 
dodgy PCIe bridges bypass the IOMMU and when the host''s virtual memory 
addresses equal the physical device''s BAR, everything crashes
spectacularly.

But I''m glad you don''t have hardware as broken as me. :)
> My domU on XP 64 was locked up (frozen) when game was starting. I had to
> change my (OR mask 1) value for soft straps so this issue went away.
> Whatever driver used it works like a charm.
Hang on - you mean to say that you managed to get VGA output on an 
external monitor with VGA passthrough WITHOUT quadrifying the card? 
Before I quadrified mine I couldn''t get anything at all - VNC console 
only, and device manager was saying it couldn''t start the device.

It makes me wonder if Nvidia are now tweaking the GeForce drivers to 
work better to discourage people from looking at modifying cards from 
GeForce to Quadro (since VGA passthrough is _the_ main benefit). Maybe 
they are concerned about what else we might figure out. ;)
> I set up a new VM for Win7 64 with RAM=8GB (no patch for Xen!). I was
> able to play ''Remember me'' :)
>
> Here is a screenshot
>
>
http://img15.hostingpics.net/pics/654044ScreenshotXenSoftmodedQuadro6000Win764.png
Good, good. Did you make sure you changed the PCI device ID in BIOS at 
around 0x18E ? This should be consistent with the PCI device ID.
> I am working on testing games on Wine with (Xen PVONHVM+NVIDIA).
>
>
> I have great esteem  for the work you did/are currently doing.
Won''t be
> back to Xen/NVIDIA world without you :)
I''m afraid I cannot take that much credit - to paraphrase the old
quote,
I have only succeeded at this by "standing on the shoulders of giants"
that reverse engineered bits of this before me.
> "GTX 500 series + soldering iron" are for professional guys like
you :).
Ironically, GTX580 is somewhat pointless to modify. You can soft-mod it 
to a Quadro 7000, but that isn''t MultiOS, so VGA passthrough
doesn''t
work (having said that - Quadro K5000 isn''t MultiOS either but VGA 
passthrough works great). There is no SPEC boost from modding the 580, 
and the 580 has no bidirectional DMA engine to enable. So the only 
narrow use-case for modding it is for some professional applications (I 
seem to recall some Adobe apps were mentioned ages ago for refusing to 
even acknowledge the presence of a GeForce card for GPU processing 
offload, but worked fine with Quadros, so in that instance there may be 
  benefit (then again, I seem to recall the same could be achieved with 
an .inf mod without modifying the BIOS at all - sadly that doesn''t seem
to work for VGA passthrough).
> Got a GTX 680 at home. I will wait for your next article on your web
> site before
OK. That can be soft-modded to a Tesla K10, but requires a hard-mod for 
K5000/K2 conversion. So you''ll have to get your soldering iron out for 
it, unfortunately. The process is well documented on the eevblog forum. 
What I can bring to the table is BIOS modding to get a full K5000 BIOS 
onto a 4GB GTX680 (will probably work just the same on a GTX770 since 
they are virtually identical). What I might work on next is stripping 
out all the UEFI wrappers so the real BIOS fits in under the 64KB limit 
for primary gfx passthrough. :)
> Please keep us posted about your investigations for e820_host for HVMs
You can keep track of it on the xen-devel list, but I haven''t had a 
chance to do anything over and above the proof of concept patches yet.

Gordan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

On Wed, 18 Sep 2013 14:15:32 +0100 (BST), David TECHER 
 <davidtecher@yahoo.fr> wrote:
> (or mask 1): updated so my issue on my domU ( XP 64) went away -- 
> when
> Crysis 2 was starting and was locking up --.
>
> Don''t ask me why it fixed my issue. I decided to modify the (or
mask
> 1) knowing that it could break something for my card. Let''s say
that
> I
> was lucky :)
>
> I just modified both (or mak 0) and (or mask 1). I just noticed that
> it works. It is enough for me.
 Can you just clarify something here - you got the GTX480 to work
 with secondary passthrough on a VM with GeForce drivers BEFORE you
 modified it into a Quadro? Or am I misunderstanding this?

 I never managed an unmodified GeForce card to work in a VM.
> Playing Windows games on a Linux domU using Wine could be problematic
> with a such card. This quadrified card is a GTX 480 -- well known to
> have temperature problems --.
 Really? I never noticed any such issues.
> At home I have two GTX 480 cards.
>
> 1) #1 is used actually for my current tests
>
> 2) #2 with _Arctic Cooling Accelero XTREME Plus._ I will try to do 
> the
> same tests on this one. This one should fixed temperature problems :)
 Odd. Mine is a standard reference design and I hadn''t noticed
 temperature issues when running Crysis at 3840x2400, which ought to
 have exercised it quite thoroughly. And that''s with the GTX470/Q5000
 in the slot right next to it.
> ------------------
> For e820_host: I am reading your mails on xen-devel list so I am
> updated :)
>
> For GTX 680/GTX 770: it is clear that the 64KB limit is the first
> problem -- primary gfx passthrough --. If you succeeded then keep us
> posted.
 Will do.

 Gordan

>Can you just clarify something here - you got the GTX480 to work
>with secondary passthrough on a VM with GeForce drivers BEFORE you
>modified it into a Quadro? Or am I misunderstanding this?
 No I was not able to do that. 


For temperature problem: 


- VM = Windows XP, Windows 7. There is no temperature problem while playing
native Windows games. I played Crysis 2 and no temperature problem.


- VM = Linux. This issue occured with this card when playing native Windows
games using Wine.

So it occured only for Linux (VM)




________________________________
 De : Gordan Bobic <gordan@bobich.net>
À : David TECHER <davidtecher@yahoo.fr> 
Cc : xen-users@lists.xen.org 
Envoyé le : Mercredi 18 septembre 2013 16h06
Objet : Re: [Xen-users] Xen VGA Passthrough - GTX 480 successfully quadrified to
quadro 6000 (softmod) - more than 4GB of RAM for Win XP 64 Bits
 

On Wed, 18 Sep 2013 14:15:32 +0100 (BST), David TECHER 
<davidtecher@yahoo.fr> wrote:
> (or mask 1): updated so my issue on my domU ( XP 64) went away -- 
> when
> Crysis 2 was starting and was locking up --.
>
> Don''t ask me why it fixed my issue. I decided to modify the (or
mask
> 1) knowing that it could break something for my card. Let''s say
that
> I
> was lucky :)
>
> I just modified both (or mak 0) and (or mask 1). I just noticed that
> it works. It is enough for me.
Can you just clarify something here - you got the GTX480 to work
with secondary passthrough on a VM with GeForce drivers BEFORE you
modified it into a Quadro? Or am I misunderstanding this?

I never managed an unmodified GeForce card to work in a VM.
> Playing Windows games on a Linux domU using Wine could be problematic
> with a such card. This quadrified card is a GTX 480 -- well known to
> have temperature problems --.
Really? I never noticed any such issues.
> At home I have two GTX 480 cards.
>
> 1) #1 is used actually for my current tests
>
> 2) #2 with _Arctic Cooling Accelero XTREME Plus._ I will try to do 
> the
> same tests on this one. This one should fixed temperature problems :)
Odd. Mine is a standard reference design and I hadn''t noticed
temperature issues when running Crysis at 3840x2400, which ought to
have exercised it quite thoroughly. And that''s with the GTX470/Q5000
in the slot right next to it.
> ------------------
> For e820_host: I am reading your mails on xen-devel list so I am
> updated :)
>
> For GTX 680/GTX 770: it is clear that the 64KB limit is the first
> problem -- primary gfx passthrough --. If you succeeded then keep us
> posted.
Will do.

Gordan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

On Wed, 18 Sep 2013 15:35:44 +0100 (BST), David TECHER 
 <davidtecher@yahoo.fr> wrote:
>>Can you just clarify something here - you got the GTX480 to work
>  >with secondary passthrough on a VM with GeForce drivers BEFORE you
>  >modified it into a Quadro? Or am I misunderstanding this?
>
>  No I was not able to do that.
 In that case:

 1) What modification had you already applied when Crysis crashed at 
 startup?
 2) What further modification was required to fix the problem?
> For temperature problem:
>
> - VM = Windows XP, Windows 7. There is no temperature problem while
> playing native Windows games. I played Crysis 2 and no temperature
> problem.
>
> - VM = Linux. This issue occured with this card when playing native
> Windows games using Wine.
>
> So it occured only for Linux (VM)
 Aha! Now I understand! So either the fan control in Linux is broken
 or the Windows driver does some temperature management throttling
 that the Linux driver doesn''t. Have you tried manually setting
 the fan speed to maximum? I could have sworn there were Linux
 utilities for this ages ago, but I haven''t really kept up with
 things because I haven''t looked into it recently.

 What temperature were you hitting on the GPU? IIRC Nvidia GPUs
 are rated to 95C, and only throttle after that (or at least that
 was the case back when I was running number-crunching on multiple
 9800GX2 cards.

 Gordan

First time I used ''nvflash --straps'' I just changed my (or
mask 0) value. So only one hex value was updated!

After that I played Crysis. Crysis crashed at startup

2nd time I reused the command ''nvflash --straps'' setting (or
mask 1) value to its new value.  It fixed the problem for Crysis

I should be able to do a few tests for Linux when I am back to home.




________________________________
 De : Gordan Bobic <gordan@bobich.net>
À : David TECHER <davidtecher@yahoo.fr> 
Cc : xen-users@lists.xen.org 
Envoyé le : Mercredi 18 septembre 2013 16h50
Objet : Re: [Xen-users] Xen VGA Passthrough - GTX 480 successfully quadrified to
quadro 6000 (softmod) - more than 4GB of RAM for Win XP 64 Bits
 

On Wed, 18 Sep 2013 15:35:44 +0100 (BST), David TECHER 
<davidtecher@yahoo.fr> wrote:
>>Can you just clarify something here - you got the GTX480 to work
>  >with secondary passthrough on a VM with GeForce drivers BEFORE you
>  >modified it into a Quadro? Or am I misunderstanding this?
>
>  No I was not able to do that.
In that case:

1) What modification had you already applied when Crysis crashed at 
startup?
2) What further modification was required to fix the problem?
> For temperature problem:
>
> - VM = Windows XP, Windows 7. There is no temperature problem while
> playing native Windows games. I played Crysis 2 and no temperature
> problem.
>
> - VM = Linux. This issue occured with this card when playing native
> Windows games using Wine.
>
> So it occured only for Linux (VM)
Aha! Now I understand! So either the fan control in Linux is broken
or the Windows driver does some temperature management throttling
that the Linux driver doesn''t. Have you tried manually setting
the fan speed to maximum? I could have sworn there were Linux
utilities for this ages ago, but I haven''t really kept up with
things because I haven''t looked into it recently.

What temperature were you hitting on the GPU? IIRC Nvidia GPUs
are rated to 95C, and only throttle after that (or at least that
was the case back when I was running number-crunching on multiple
9800GX2 cards.

Gordan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

On Wed, 18 Sep 2013 16:08:04 +0100 (BST), David TECHER 
 <davidtecher@yahoo.fr> wrote:
> First time I used 'nvflash --straps' I just changed my (or mask 0)
> value. So only one hex value was updated!
>
> After that I played Crysis. Crysis crashed at startup
>
> 2nd time I reused the command 'nvflash --straps' setting (or mask
1)
> value to its new value. It fixed the problem for Crysis
 The mask order is AND1 OR1 AND2 OR2. You shouldn't need to touch
 anything in AND2 and OR2. It is only AND1 and OR1 that contain
 the bits that you should be changing in this case. The second
 set of straps affects other things that you almost certainly
 shouldn't be changing unless you know specifically what you
 are aiming to do.

 So I cannot quite figure out what you are referring to here. :(

 Gordan
> I should be able to do a few tests for Linux when I am back to home.
>
> -------------------------
>   DE : Gordan Bobic
>  À : David TECHER
> CC : xen-users@lists.xen.org
>  ENVOYÉ LE : Mercredi 18 septembre 2013 16h50
>  OBJET : Re: [Xen-users] Xen VGA Passthrough - GTX 480 successfully
> quadrified to quadro 6000 (softmod) - more than 4GB of RAM for Win XP
> 64 Bits
>
>  On Wed, 18 Sep 2013 15:35:44 +0100 (BST), David TECHER
>   wrote:
>>>Can you just clarify something here - you got the GTX480 to work
>> >with secondary passthrough on a VM with GeForce drivers BEFORE you
>> >modified it into a Quadro? Or am I misunderstanding this?
>>
>> No I was not able to do that.
>
>  In that case:
>
>  1) What modification had you already applied when Crysis crashed at
>  startup?
>  2) What further modification was required to fix the problem?
>
>> For temperature problem:
>>
>> - VM = Windows XP, Windows 7. There is no temperature problem while
>> playing native Windows games. I played Crysis 2 and no temperature
>> problem.
>>
>> - VM = Linux. This issue occured with this card when playing native
>> Windows games using Wine.
>>
>> So it occured only for Linux (VM)
>
>  Aha! Now I understand! So either the fan control in Linux is broken
>  or the Windows driver does some temperature management throttling
>  that the Linux driver doesn't. Have you tried manually setting
>  the fan speed to maximum? I could have sworn there were Linux
>  utilities for this ages ago, but I haven't really kept up with
>  things because I haven't looked into it recently.
>
>  What temperature were you hitting on the GPU? IIRC Nvidia GPUs
>  are rated to 95C, and only throttle after that (or at least that
>  was the case back when I was running number-crunching on multiple
>  9800GX2 cards.
>
>  Gordan
>
>
>
> Links:
> ------
> [1] mailto:davidtecher@yahoo.fr

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

AND1 OR1 AND2 OR2.

Yes.  This is the correct order.


OR1 and OR2 have been updated in my case. I didn''t change AND1and AND2.

To be honest I am lucky that my card is still working.



________________________________
 De : Gordan Bobic <gordan@bobich.net>
À : David TECHER <davidtecher@yahoo.fr> 
Cc : xen-users@lists.xen.org 
Envoyé le : Mercredi 18 septembre 2013 17h13
Objet : Re: [Xen-users] Xen VGA Passthrough - GTX 480 successfully quadrified to
quadro 6000 (softmod) - more than 4GB of RAM for Win XP 64 Bits
 

On Wed, 18 Sep 2013 16:08:04 +0100 (BST), David TECHER 
<davidtecher@yahoo.fr> wrote:
> First time I used ''nvflash --straps'' I just changed my
(or mask 0)
> value. So only one hex value was updated!
>
> After that I played Crysis. Crysis crashed at startup
>
> 2nd time I reused the command ''nvflash --straps'' setting
(or mask 1)
> value to its new value. It fixed the problem for Crysis
The mask order is AND1 OR1 AND2 OR2. You shouldn''t need to touch
anything in AND2 and OR2. It is only AND1 and OR1 that contain
the bits that you should be changing in this case. The second
set of straps affects other things that you almost certainly
shouldn''t be changing unless you know specifically what you
are aiming to do.

So I cannot quite figure out what you are referring to here. :(

Gordan
> I should be able to do a few tests for Linux when I am back to home.
>
> -------------------------
>   DE : Gordan Bobic
>  À : David TECHER
> CC : xen-users@lists.xen.org
>  ENVOYÉ LE : Mercredi 18 septembre 2013 16h50
>  OBJET : Re: [Xen-users] Xen VGA Passthrough - GTX 480 successfully
> quadrified to quadro 6000 (softmod) - more than 4GB of RAM for Win XP
> 64 Bits
>
>  On Wed, 18 Sep 2013 15:35:44 +0100 (BST), David TECHER
>   wrote:
>>>Can you just clarify something here - you got the GTX480 to work
>> >with secondary passthrough on a VM with GeForce drivers BEFORE you
>> >modified it into a Quadro? Or am I misunderstanding this?
>>
>> No I was not able to do that.
>
>  In that case:
>
>  1) What modification had you already applied when Crysis crashed at
>  startup?
>  2) What further modification was required to fix the problem?
>
>> For temperature problem:
>>
>> - VM = Windows XP, Windows 7. There is no temperature problem while
>> playing native Windows games. I played Crysis 2 and no temperature
>> problem.
>>
>> - VM = Linux. This issue occured with this card when playing native
>> Windows games using Wine.
>>
>> So it occured only for Linux (VM)
>
>  Aha! Now I understand! So either the fan control in Linux is broken
>  or the Windows driver does some temperature management throttling
>  that the Linux driver doesn''t. Have you tried manually setting
>  the fan speed to maximum? I could have sworn there were Linux
>  utilities for this ages ago, but I haven''t really kept up with
>  things because I haven''t looked into it recently.
>
>  What temperature were you hitting on the GPU? IIRC Nvidia GPUs
>  are rated to 95C, and only throttle after that (or at least that
>  was the case back when I was running number-crunching on multiple
>  9800GX2 cards.
>
>  Gordan
>
>
>
> Links:
> ------
> [1] mailto:davidtecher@yahoo.fr
_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

On Wed, 18 Sep 2013 16:39:26 +0100 (BST), David TECHER 
 <davidtecher@yahoo.fr> wrote:
> AND1 OR1 AND2 OR2.
>
> Yes. This is the correct order.
>
> OR1 and OR2 have been updated in my case. I didn''t change AND1 and
> AND2.
>
> To be honest I am lucky that my card is still working.
 It certainly sounds like it. So the first time you correctly
 toggled bit 13 in the OR1 mask to 1. That should be all that
 was required.

 Then you toggled up bit 13 in OR2 mask to 1 - right?

 I don''t know what that bit does - I haven''t been able to
identify
 it''s purpose. It might be worth changing it back and verifying
 that the crash you originally saw wasn''t just a random occurrence.

 I should probably try to re-check what those bits do. There might
 be something identifiably useful there.

 Gordan

I reset OR2 to its default value. 

I did a test and my VM Win7-64 and my VM Linux-64 work. My VM Win7-64 is a
fresh/new VM. Everything works fine for this VM.

I think that my VM WinXP-64 is broken. This is an old VM (I guess that something
is broken for this VM). I will rebuid a new domU WinXP-64 in a couple of days
and will let you know.

VM WinXP-64 is an old VM so uninstalling/installing NVIDIA drivers, Xen gp lpv
drivers into this VM should broke something. I don''t think that my
issue for this VM is related to the card BIOS.

Keep you posted.

Thanks




________________________________
 De : Gordan Bobic <gordan@bobich.net>
À : David TECHER <davidtecher@yahoo.fr> 
Cc : xen-users@lists.xen.org 
Envoyé le : Mercredi 18 septembre 2013 18h18
Objet : Re: [Xen-users] Xen VGA Passthrough - GTX 480 successfully quadrified to
quadro 6000 (softmod) - more than 4GB of RAM for Win XP 64 Bits
 

On Wed, 18 Sep 2013 16:39:26 +0100 (BST), David TECHER 
<davidtecher@yahoo.fr> wrote:
> AND1 OR1 AND2 OR2.
>
> Yes. This is the correct order.
>
> OR1 and OR2 have been updated in my case. I didn''t change AND1 and
> AND2.
>
> To be honest I am lucky that my card is still working.
It certainly sounds like it. So the first time you correctly
toggled bit 13 in the OR1 mask to 1. That should be all that
was required.

Then you toggled up bit 13 in OR2 mask to 1 - right?

I don''t know what that bit does - I haven''t been able to
identify
it''s purpose. It might be worth changing it back and verifying
that the crash you originally saw wasn''t just a random occurrence.

I should probably try to re-check what those bits do. There might
be something identifiably useful there.

Gordan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

Finaly I decided to rebuild a domU WinXP 64 today from scratch.

So the card works as expected.  Here is my summary

domU = WinXP 64: RAM is limited to 3GB but the domU works perfecty. Setting RAM
over 3GB the domU gets slower.


domU = Win7 64. RAM is not limited to 3GB. Can set more than 3GB


domU = Linux 32/64: RAM is not limited. Can set more than 3GB.

I will stop my tests here. Thanks

Kind regards.


________________________________
 De : David TECHER <davidtecher@yahoo.fr>
À : Gordan Bobic <gordan@bobich.net> 
Cc : "xen-users@lists.xen.org" <xen-users@lists.xen.org> 
Envoyé le : Jeudi 19 septembre 2013 1h34
Objet : Re: [Xen-users] Xen VGA Passthrough - GTX 480 successfully quadrified to
quadro 6000 (softmod) - more than 4GB of RAM for Win XP 64 Bits
 



I reset OR2 to its default value. 

I did a test and my VM Win7-64 and my VM Linux-64 work. My VM Win7-64 is a
fresh/new VM. Everything works fine for this VM.

I think that my VM WinXP-64 is broken. This is an old VM (I guess that something
is broken for this VM). I will rebuid a new domU WinXP-64 in a couple of days
and will let you know.

VM WinXP-64 is an old VM so uninstalling/installing NVIDIA drivers, Xen gp lpv
drivers into this VM should broke something. I don''t think that my
issue for this VM is related to the card BIOS.

Keep you posted.

Thanks




________________________________
 De : Gordan Bobic <gordan@bobich.net>
À : David TECHER <davidtecher@yahoo.fr> 
Cc : xen-users@lists.xen.org 
Envoyé le : Mercredi 18 septembre 2013 18h18
Objet : Re: [Xen-users] Xen VGA Passthrough - GTX 480 successfully quadrified to
quadro 6000 (softmod) - more than 4GB of RAM for Win XP 64 Bits
 

On Wed, 18 Sep 2013 16:39:26 +0100 (BST), David TECHER 
<davidtecher@yahoo.fr> wrote:
> AND1 OR1 AND2 OR2.
>
> Yes. This is the correct order.
>
> OR1 and OR2 have been
 updated in my case. I didn''t change AND1 and
> AND2.
>
> To be honest I am lucky that my card is still working.
It certainly sounds like it. So the first time you correctly
toggled bit 13 in the OR1 mask to 1. That should be all that
was required.

Then you toggled up bit 13 in OR2 mask to 1 - right?

I don''t know what that bit does - I haven''t been able to
identify
it''s purpose. It might be worth changing it back and verifying
that the crash you originally saw wasn''t just a random occurrence.

I should probably try to re-check what those bits do. There might
be something identifiably useful there.

Gordan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

On Sun, 22 Sep 2013 18:43:55 +0100 (BST), David TECHER 
 <davidtecher@yahoo.fr> wrote:
> Finaly I decided to rebuild a domU WinXP 64 today from scratch.
>
> So the card works as expected. Here is my summary
>
> domU = WinXP 64: RAM is limited to 3GB but the domU works perfecty.
> Setting RAM over 3GB the domU gets slower.
 Can you elaborate on this? I am now running XP64 with > 5GB of RAM
 (needed to apply my work-in-progres e820_host HVM patch) and
 I haven''t noticed any slowdown compared to before when running
 with 2.5GB of RAM in domU. How did you test/observe the slowdown?

 Gordan

FYI I didn''t try your patch for the moment.

I noticed the slowdown when setting RAM > 3GB in configuration file. I did
the test on a old and fresh current VM XP64.

Yesterday I set up a new VM XP 64 from scratch. Once NVIDIA driver was installed
I restarted the VM. Extremely slow if RAM > 3GB.

So I set RAM <= 3GB VM worked pretty fine


Perhaps I may have a try for your patch and let you know.

David



________________________________
 De : Gordan Bobic <gordan@bobich.net>
À : xen-users@lists.xen.org 
Envoyé le : Lundi 23 septembre 2013 13h12
Objet : Re: [Xen-users] Xen VGA Passthrough - GTX 480 successfully quadrified to
quadro 6000 (softmod) - more than 4GB of RAM for Win XP 64 Bits
 

On Sun, 22 Sep 2013 18:43:55 +0100 (BST), David TECHER 
<davidtecher@yahoo.fr> wrote:
> Finaly I decided to rebuild a domU WinXP 64 today from scratch.
>
> So the card works as expected. Here is my summary
>
> domU = WinXP 64: RAM is limited to 3GB but the domU works perfecty.
> Setting RAM over 3GB the domU gets slower.
Can you elaborate on this? I am now running XP64 with > 5GB of RAM
(needed to apply my work-in-progres e820_host HVM patch) and
I haven''t noticed any slowdown compared to before when running
with 2.5GB of RAM in domU. How did you test/observe the slowdown?

Gordan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users

On 09/23/2013 07:33 PM, David TECHER wrote:
> FYI I didn''t try your patch for the moment.
My patch is _only_ needed on broken hardware where the DMA bypasses the 
IOMMU, e.g. with NF200 PCIe bridges, and while it is possible that other 
hardware is affected by similar bugs, I have not heard reports to that 
effect.

And for the record, the irony is not lost on me of Nvidia NF200 bridges 
thoroughly breaking virtualization while Nvidia GPUs being the only 
fit-for-purpose option for VGA passthrough.
> I noticed the slowdown when setting RAM > 3GB in configuration file. I
> did the test on a old and fresh current VM XP64.
Can you explain what do you mean by slowdown? I''m wondering if this is 
in some way related to memory hole remapping.
> Yesterday I set up a new VM XP 64 from scratch. Once NVIDIA driver was
> installed I restarted the VM. Extremely slow if RAM > 3GB.
Just out of interest - what do you get from:
lspci -vvv | grep "Region " | grep "Memory at " | sed -e
''s/.*Memory at
//'' | sort

Can you check if the slowdown begins when you assign the amount of 
memory to the domU that begins to exceed the address at the first line 
of that command?
> So I set RAM <= 3GB VM worked pretty fine
>
> Perhaps I may have a try for your patch and let you know.
The last patch (WORK IN PROGRESS!) I posted was specific to my setup, 
and IIRC it essentially marks everything above 1GB as a memory hole, but 
it didn''t adjust the top end of the memory, so you effectively lose 
2.5GB of RAM you allocate to the domU. By all means try it, just bear in 
mind it is not for for general consumption.

Gordan