Xen users - May 2011 - nvidia drivers do not seem to work on xen kernels

Hi guys,

I spent the whole week trying to setup a Ubuntu 10.04 64-bit machine for
Xen virtualization (Xen 4.1) for GPGPU tests on several virtual machines
(using Xen''s GPU passthrough capability). My problem is that I just
can''t get the nvidia development driver to work on the xen kernel
(works
perfectly fine with the standard kernel). 

For compiling the kernel, I followed the instructions from
http://www.zeroaccess.org/2011/04/xen-4-1-on-ubuntu-10-04-64bit/ (but
built the kernel the debian way (make-kpkg) to get a nice .deb package).
It boots fine, I modified grub to set the kernel options, etc. All the
Xen setup seems to be working as it should. 

I can''t get the NVIDIA driver working on the host (Dom0) though. I used
the driver with CUDA 3.2, and also the 4.0 RC2 driver (for a GTX 590
card). When the system tries to start X11, the screen turns blank and
the system gets very slow. Booting the system into text mode works fine.
I can load the nvidia driver manually (modprobe nvidia), and create the
device nodes in /dev using mknode (so I have
the /dev/nvidia0, /dev/nvidia1, /dev/nvidiactl - with major number 195,
and minors 0, 1, 255, respectively). When I try to build anything using
OpenCL, it just reports that no platforms have been found. With CUDA, I
get the error: "cudaSafeCall() Runtime API error : invalid device
ordinal." Both work completely fine when I boot the system into a
standard kernel (linux-image-generic, default with ubuntu 10.04).

The X11 log just says that it failed to load the NVIDIA module. Syslog
gives messages like "NVRM: RmInitAdapter failed!" .

I tried various suggestions for installing the the driver found on the
web (e.g., http://wiki.xensource.com/xenwiki/NvidiaGPU?highlight=%
28nvidia%29 ) but with no success.

Did any of you get NVIDIA and Xen work together?? How? Any help is
appreciated!

Jorg



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

>>I tried various suggestions for installing the the driver found on the
>>web (e.g., http://wiki.xensource.com/xenwiki/NvidiaGPU?highlight=%
>>28nvidia%29 ) but with no success.
>>
>>Did any of you get NVIDIA and Xen work together?? How? Any help is
>>appreciated!
>>Jorg
Hi,

Did you see the table ?
xen/stable-* pvops branches
All versions work properly on native. The module will build and load
successfully
on bare-metal (e.g. without the hypervisor), but will just present a
BLANK SCREEN
under Xen, without any detectable errors.

I tried to run Nvidia under Debian Squeeze but the result was the same
as mentioned above.

There is a bug report for Debian:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=570365
And here is an interesting thread:
http://lists.debian.org/debian-kernel/2010/10/msg00289.html

I think there is few people who need to run Nvidia drivers under Xen,
that''s why it is still unfixed.

Regards,


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, May 06, 2011 at 04:06:41PM +0100, Jorg Lotze
wrote:
> Hi guys,
> 
> I spent the whole week trying to setup a Ubuntu 10.04 64-bit machine for
> Xen virtualization (Xen 4.1) for GPGPU tests on several virtual machines
> (using Xen''s GPU passthrough capability). My problem is that I
just
> can''t get the nvidia development driver to work on the xen kernel
(works
> perfectly fine with the standard kernel). 
> 
> For compiling the kernel, I followed the instructions from
> http://www.zeroaccess.org/2011/04/xen-4-1-on-ubuntu-10-04-64bit/ (but
> built the kernel the debian way (make-kpkg) to get a nice .deb package).
> It boots fine, I modified grub to set the kernel options, etc. All the
> Xen setup seems to be working as it should. 
> 
> I can''t get the NVIDIA driver working on the host (Dom0) though. I
used
> the driver with CUDA 3.2, and also the 4.0 RC2 driver (for a GTX 590
> card). When the system tries to start X11, the screen turns blank and
> the system gets very slow. Booting the system into text mode works fine.
> I can load the nvidia driver manually (modprobe nvidia), and create the
> device nodes in /dev using mknode (so I have
> the /dev/nvidia0, /dev/nvidia1, /dev/nvidiactl - with major number 195,
> and minors 0, 1, 255, respectively). When I try to build anything using
> OpenCL, it just reports that no platforms have been found. With CUDA, I
> get the error: "cudaSafeCall() Runtime API error : invalid device
> ordinal." Both work completely fine when I boot the system into a
> standard kernel (linux-image-generic, default with ubuntu 10.04).
> 
> The X11 log just says that it failed to load the NVIDIA module. Syslog
> gives messages like "NVRM: RmInitAdapter failed!" .
> 
> I tried various suggestions for installing the the driver found on the
> web (e.g., http://wiki.xensource.com/xenwiki/NvidiaGPU?highlight=%
> 28nvidia%29 ) but with no success.
> 
> Did any of you get NVIDIA and Xen work together?? How? Any help is
> appreciated!
> 
IIRC there was some environment/cmdline option in nvidia drivers
to bypass some checks under Xen.. ?

-- Pasi


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> IIRC there was some environment/cmdline option in nvidia drivers
> to bypass some checks under Xen.. ?
Yes, there is, the problem is that the drivers actually don''t work this
time. I have the same problem on OpenSuSe 11.3.

The drivers shouldn''t have worked even before, but usually they did,
now
they don''t.

-- 
Mgr. Simon Toth

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> IIRC there was some environment/cmdline option in nvidia drivers
> to bypass some checks under Xen.. ?
>Yes, there is, the problem is that the drivers actually don''t work
this
>time. I have the same problem on OpenSuSe 11.3.
>
>The drivers shouldn''t have worked even before, but usually they
did, now
>they don''t.
>--
>Mgr. Simon Toth

Some people says that it works on OpenSuSe 11.3 and 11.4:
http://forums.nvidia.com/index.php?showtopic=199735&st=0&p=1234631&hl=dom0&fromsearch=1&#entry1234631
There is also a link:
http://old-en.opensuse.org/Talk:Use_Nvidia_driver_with_Xen

Could you check it ?


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

I''m also searching the solution of what you''ve asked here. 

I tried two kinds of xen environment 

1. Xen 4.0.2 + 2.6.32 pvops dom0 kernel
 -> all apps failed.
 -> error message (dmesg), NVRM: PAT Configuration unsupported + NVRM:
RmInitAdapter failed! (0x27:0x38:1047)
 -> app error message, /dev/nvidia0 io failed

2. Xen 3.2 + 2.6.18 (centos 5.5) dom0 kernel
 -> device query is possible, while bandwidth test failed
 -> error message (dmesg), NVRM: bad caching on address 0xffff8805b62cd000:
actual 0x77 != expected 0x73
 -> app error message, cudasafecall runtime api error : all CUDA-available
device is busy or unavailable



Do we need special kernel options for it? (especially related with PCI such as
Passthrough or VPCI ..)

I also spent the whole week... 

I''m ready to do anything from formatting the disk to meeting the
devil..

Please, help me out! 


>-----Original Message-----
>From: xen-users-bounces@lists.xensource.com [mailto:xen-users-
>bounces@lists.xensource.com] On Behalf Of Jorg Lotze
>Sent: Saturday, May 07, 2011 12:07 AM
>To: xen-users@lists.xensource.com
>Subject: [Xen-users] nvidia drivers do not seem to work on xen kernels
>
>Hi guys,
>
>I spent the whole week trying to setup a Ubuntu 10.04 64-bit machine for
>Xen virtualization (Xen 4.1) for GPGPU tests on several virtual machines
>(using Xen''s GPU passthrough capability). My problem is that I just
can''t
>get the nvidia development driver to work on the xen kernel (works
>perfectly fine with the standard kernel).
>
>For compiling the kernel, I followed the instructions from
>http://www.zeroaccess.org/2011/04/xen-4-1-on-ubuntu-10-04-64bit/ (but built
>the kernel the debian way (make-kpkg) to get a nice .deb package).
>It boots fine, I modified grub to set the kernel options, etc. All the Xen
>setup seems to be working as it should.
>
>I can''t get the NVIDIA driver working on the host (Dom0) though. I
used the
>driver with CUDA 3.2, and also the 4.0 RC2 driver (for a GTX 590 card).
>When the system tries to start X11, the screen turns blank and the system
>gets very slow. Booting the system into text mode works fine.
>I can load the nvidia driver manually (modprobe nvidia), and create the
>device nodes in /dev using mknode (so I have the /dev/nvidia0, /dev/nvidia1,
>/dev/nvidiactl - with major number 195, and minors 0, 1, 255, respectively).
>When I try to build anything using OpenCL, it just reports that no
>platforms have been found. With CUDA, I get the error: "cudaSafeCall()
>Runtime API error : invalid device ordinal." Both work completely fine
when
>I boot the system into a standard kernel (linux-image-generic, default with
>ubuntu 10.04).
>
>The X11 log just says that it failed to load the NVIDIA module. Syslog
>gives messages like "NVRM: RmInitAdapter failed!" .
>
>I tried various suggestions for installing the the driver found on the web
>(e.g., http://wiki.xensource.com/xenwiki/NvidiaGPU?highlight=%
>28nvidia%29 ) but with no success.
>
>Did any of you get NVIDIA and Xen work together?? How? Any help is
>appreciated!
>
>Jorg


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

>>Do we need special kernel options for it? (especially related with PCI
>>such as Passthrough or VPCI ..)
>>I also spent the whole week...
>>I''m ready to do anything from formatting the disk to meeting
the  devil..
>>Please, help me out!
I think it won''t help much because this case requires support from
programmers'' side,
especially those from Nvidia. Am I right ?

I planned to build my htpc  based on XEN but in the end moved to Vserver.
http://blog.ezrodlo.pl/tech/asus-at3iont-i-deluxe-debian-wszystko-w-jednym/


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, 2011-05-17 at 11:35 +0200, ubik pl wrote:
> >Yes, there is, the problem is that the drivers actually don''t
work
> this
> >time. I have the same problem on OpenSuSe 11.3.
> >
> >The drivers shouldn''t have worked even before, but usually
they did,
> now
> >they don''t.
> >--
> >Mgr. Simon Toth
> 
> 
> Some people says that it works on OpenSuSe 11.3 and 11.4:
>
http://forums.nvidia.com/index.php?showtopic=199735&st=0&p=1234631&hl=dom0&fromsearch=1&#entry1234631
> There is also a link:
> http://old-en.opensuse.org/Talk:Use_Nvidia_driver_with_Xen
> 
> Could you check it ? 
That''s what I did - I installed a fresh OpenSuSE 11.4 64bit and tried
to
follow these instructions. Still not working though. I got the NVIDIA
drivers installed with no problem, I even can run the Xserver correctly
under the Dom0 and everything works. However, if I try to run any of the
OpenCL/CUDA examples or my own code, it reports a CL_OUT_OF_RESOURCES
error, or that all devices are busy. 

(the Nvidia forum post up you linked in your reply is actually mine -
donnervogel)

Ideas, anyone?

J



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Ideas, anyone?
Make sure that your Xserver is actually running with the Nvidia driver 
not the nv or Noveau driver. Mine did start up with Noveau and Nvidia 
kept failing.

-- 
Mgr. Simon Toth



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Ideas, anyone?
Make sure that your Xserver is actually running with the Nvidia driver 
not the nv or Noveau driver. Mine did start up with Noveau and Nvidia 
kept failing.

-- 
Mgr. Simon Toth

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, 2011-05-18 at 15:27 +0200, "Mgr. Šimon Tóth"
wrote:
> Make sure that your Xserver is actually running with the Nvidia driver 
> not the nv or Noveau driver. Mine did start up with Noveau and Nvidia 
> kept failing.
> 
Did that - blacklisted noveau. Also started without X, loaded the nvidia
module myself, and created the device nodes with the right permission
myself - didn''t work either. So it''s not that...

J



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

>>Did that - blacklisted noveau. Also started without X, loaded the nvidia
>>module myself, and created the device nodes with the right permission
>>myself - didn''t work either. So it''s not that...
Read this: http://wiki.qubes-os.org/trac/wiki/InstallNvidiaDriver

cat /etc/modprobe.d/nouveau-disable.conf
# blacklist isn''t enough...
install nouveau /bin/true

Add *rdblacklist=nouveau* option to /boot/grub/menu.lst (at the end of line
containing *vmlinuz*).


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, 2011-05-18 at 16:49 +0200, ubik pl wrote:
> Read this: http://wiki.qubes-os.org/trac/wiki/InstallNvidiaDriver
> 
> cat /etc/modprobe.d/nouveau-disable.conf
> # blacklist isn''t enough...
> install nouveau /bin/true
> 
> Add rdblacklist=nouveau option to /boot/grub/menu.lst (at the end of
> line containing vmlinuz).
I am really sure that noveau isn''t loaded. First, nvidia-settings in X
reports that the nvidia driver is used. Second, lsmod | grep noveau
shows nothing. Third, everything works fine when I boot the exact same
system with a non-xen kernel, including the CUDA/OpenCL tests. So this
is not the issue.

J




_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

>-----Original Message-----
>From: xen-users-bounces@lists.xensource.com [mailto:xen-users-
>bounces@lists.xensource.com] On Behalf Of Jorg Lotze
>Sent: Thursday, May 19, 2011 12:10 AM
>To: xen-users@lists.xensource.com
>Subject: Re: [Xen-users] nvidia drivers do not seem to work on xen kernels
>
>On Wed, 2011-05-18 at 16:49 +0200, ubik pl wrote:
>> Read this: http://wiki.qubes-os.org/trac/wiki/InstallNvidiaDriver
>>
>> cat /etc/modprobe.d/nouveau-disable.conf
>> # blacklist isn''t enough...
>> install nouveau /bin/true
>>
>> Add rdblacklist=nouveau option to /boot/grub/menu.lst (at the end of
>> line containing vmlinuz).
>
>I am really sure that noveau isn''t loaded. First, nvidia-settings
in X
>reports that the nvidia driver is used. Second, lsmod | grep noveau shows
>nothing. Third, everything works fine when I boot the exact same system
>with a non-xen kernel, including the CUDA/OpenCL tests. So this is not the
>issue.
>
I am also sure that it is not the case (nouveau). 

I installed opensuse11.4 as a last trial. 

But I''ve got same error with the trial before (centos 5.5, xen provided
by centos)

Let me elaborate little bit. 

Test...
 1. nvidia-smi works fine
 2. devicequery also works fine.
 3. except devicequery, all applications fail. Error message is same "
cudaSafeCall() Runtime API error : all CUDA-capable devices are busy or
unavailable."

Nvidia-smi & device query means it is possible to access the device
correctly.

I guess,,, the problem is cudasafecall doesn''t work at all even though
the function is used in almost all GPU programs.

I hope somebody to break this monster.. 










_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users