Hello, I''m looking to enable the NVIDIA CUDA driver/runtime stack to work in Dom0 on Xen. I''ve contacted NVIDIA through various capacities and have gotten replies essentially saying they cannot provide assistance, and have been following the nvnews.com forums. However, now that I have IOMMU-capable processors (with VT-d) and a version of Xen which can successfully program this hardware, I am interested to determine if it is possible to program the IOMMU *for* Dom0 (i.e. the target is Dom0, not an HVM/pvops guest). I have successfully been able to launch an HVM guest, passing a few GPUs through. Installing an unmodified developer driver from NVIDIA works using the standard method (no SYSSRC=/-OUT= or anything) and CUDA applications execute as expected. Can this be done for Dom0, achieving the same result without requiring any involvement from NVIDIA or modifications to the driver? My immediate interest is more to see if it "can be done" via a hack or something, not necessarily whether it would make sense for Xen to support this in the future. My goal with this email is to get feedback on two fronts: 1) is there a limitation due to the architecture, meaning that as Dom0 is pvops it cannot use VT-d, or is it a implementation addition to Xen that would be needed? 2) If the latter is true (programming needed), how much effort (code/time) would you/anyone estimate it would take to enable this to work, and could you provide some starting pointers for me to do so? I am untrained when it comes to the Xen sourcecode. I''m using Xen 4.1.1 and pv-ops linux (not upstream) 2.6.32.40 on an Intel X5660 with a Tylersburg chipset. My host OS is Fedora 13, but that needn''t be static. I originally asked this on xen-users, but was informed this mailing list would be better suited: http://lists.xensource.com/archives/html/xen-users/2011-06/msg00451.html Many thanks in advance! Alex _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello Alex, What nvidia cards did you use to succesfully passthrough to a HVM and have CUDA working ? This could be of interest for others trying to achieve the same ... -- Sander Monday, June 27, 2011, 5:35:17 AM, you wrote:> Hello,> I''m looking to enable the NVIDIA CUDA driver/runtime stack to work in > Dom0 on Xen. I''ve contacted NVIDIA through various capacities and have > gotten replies essentially saying they cannot provide assistance, and > have been following the nvnews.com forums. However, now that I have > IOMMU-capable processors (with VT-d) and a version of Xen which can > successfully program this hardware, I am interested to determine if it > is possible to program the IOMMU *for* Dom0 (i.e. the target is Dom0, > not an HVM/pvops guest).> I have successfully been able to launch an HVM guest, passing a few > GPUs through. Installing an unmodified developer driver from NVIDIA > works using the standard method (no SYSSRC=/-OUT= or anything) and > CUDA applications execute as expected. Can this be done for Dom0, > achieving the same result without requiring any involvement from > NVIDIA or modifications to the driver?> My immediate interest is more to see if it "can be done" via a hack or > something, not necessarily whether it would make sense for Xen to > support this in the future. My goal with this email is to get feedback > on two fronts: 1) is there a limitation due to the architecture, > meaning that as Dom0 is pvops it cannot use VT-d, or is it a > implementation addition to Xen that would be needed? 2) If the latter > is true (programming needed), how much effort (code/time) would > you/anyone estimate it would take to enable this to work, and could > you provide some starting pointers for me to do so? I am untrained > when it comes to the Xen sourcecode.> I''m using Xen 4.1.1 and pv-ops linux (not upstream) 2.6.32.40 on an > Intel X5660 with a Tylersburg chipset. My host OS is Fedora 13, but > that needn''t be static.> I originally asked this on xen-users, but was informed this mailing > list would be better suited: > http://lists.xensource.com/archives/html/xen-users/2011-06/msg00451.html> Many thanks in advance! > Alex-- Best regards, Sander mailto:linux@eikelenboom.it _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, Jun 27, 2011 at 00:01, Sander Eikelenboom <linux@eikelenboom.it> wrote:> Hello Alex, > > What nvidia cards did you use to succesfully passthrough to a HVM and have CUDA working ?NVIDIA Tesla ("Fermi") C2070 http://www.nvidia.com/object/personal-supercomputing.html Using the latest driver available, 270.41.19 http://developer.nvidia.com/cuda-toolkit-40#Linux Motherboard - Supermicro X8DTG-D> This could be of interest for others trying to achieve the same ...HVM pass-through you mean? I''m unsure how much demand there still is for the toolstack to work in Dom0, but I can say I''m still pushing.> > -- > Sander > > Monday, June 27, 2011, 5:35:17 AM, you wrote: > >> Hello, > >> I''m looking to enable the NVIDIA CUDA driver/runtime stack to work in >> Dom0 on Xen. I''ve contacted NVIDIA through various capacities and have >> gotten replies essentially saying they cannot provide assistance, and >> have been following the nvnews.com forums. However, now that I have >> IOMMU-capable processors (with VT-d) and a version of Xen which can >> successfully program this hardware, I am interested to determine if it >> is possible to program the IOMMU *for* Dom0 (i.e. the target is Dom0, >> not an HVM/pvops guest). > >> I have successfully been able to launch an HVM guest, passing a few >> GPUs through. Installing an unmodified developer driver from NVIDIA >> works using the standard method (no SYSSRC=/-OUT= or anything) and >> CUDA applications execute as expected. Can this be done for Dom0, >> achieving the same result without requiring any involvement from >> NVIDIA or modifications to the driver? > >> My immediate interest is more to see if it "can be done" via a hack or >> something, not necessarily whether it would make sense for Xen to >> support this in the future. My goal with this email is to get feedback >> on two fronts: 1) is there a limitation due to the architecture, >> meaning that as Dom0 is pvops it cannot use VT-d, or is it a >> implementation addition to Xen that would be needed? 2) If the latter >> is true (programming needed), how much effort (code/time) would >> you/anyone estimate it would take to enable this to work, and could >> you provide some starting pointers for me to do so? I am untrained >> when it comes to the Xen sourcecode. > >> I''m using Xen 4.1.1 and pv-ops linux (not upstream) 2.6.32.40 on an >> Intel X5660 with a Tylersburg chipset. My host OS is Fedora 13, but >> that needn''t be static. > >> I originally asked this on xen-users, but was informed this mailing >> list would be better suited: >> http://lists.xensource.com/archives/html/xen-users/2011-06/msg00451.html > >> Many thanks in advance! >> Alex > > > > > > -- > Best regards, > Sander mailto:linux@eikelenboom.it > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sun, Jun 26, 2011 at 09:35:17PM -0600, Alex Merritt wrote:> Hello, > > I''m looking to enable the NVIDIA CUDA driver/runtime stack to work in > Dom0 on Xen. I''ve contacted NVIDIA through various capacities and haveI''ve been running the NVIDIA binary driver with 2.6.39 / 3.0 kernel.> gotten replies essentially saying they cannot provide assistance, and > have been following the nvnews.com forums. However, now that I have > IOMMU-capable processors (with VT-d) and a version of Xen which can > successfully program this hardware, I am interested to determine if it > is possible to program the IOMMU *for* Dom0 (i.e. the target is Dom0, > not an HVM/pvops guest). > > I have successfully been able to launch an HVM guest, passing a few > GPUs through. Installing an unmodified developer driver from NVIDIA > works using the standard method (no SYSSRC=/-OUT= or anything) and > CUDA applications execute as expected. Can this be done for Dom0, > achieving the same result without requiring any involvement from > NVIDIA or modifications to the driver?It should be no trouble.. albeit you might need to update the Linux kernel to 2.6.39 or 3.0 to take advantage of the 1-1 P2M mapping code.> > My immediate interest is more to see if it "can be done" via a hack or > something, not necessarily whether it would make sense for Xen to > support this in the future. My goal with this email is to get feedback > on two fronts: 1) is there a limitation due to the architecture, > meaning that as Dom0 is pvops it cannot use VT-d, or is it a > implementation addition to Xen that would be needed? 2) If the latter > is true (programming needed), how much effort (code/time) would > you/anyone estimate it would take to enable this to work, and could > you provide some starting pointers for me to do so? I am untrained > when it comes to the Xen sourcecode. > > I''m using Xen 4.1.1 and pv-ops linux (not upstream) 2.6.32.40 on an > Intel X5660 with a Tylersburg chipset. My host OS is Fedora 13, but > that needn''t be static. > > I originally asked this on xen-users, but was informed this mailing > list would be better suited: > http://lists.xensource.com/archives/html/xen-users/2011-06/msg00451.html > > Many thanks in advance! > Alex > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad, On Mon, Jun 27, 2011 at 10:11, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:> On Sun, Jun 26, 2011 at 09:35:17PM -0600, Alex Merritt wrote: >> Hello, >> >> I''m looking to enable the NVIDIA CUDA driver/runtime stack to work in >> Dom0 on Xen. I''ve contacted NVIDIA through various capacities and have > > I''ve been running the NVIDIA binary driver with 2.6.39 / 3.0 kernel.To avoid confusion here, with 2.6.39* I have been able to successfully build, install and load the NVIDIA binary driver. Executing the SDK applications fail (except for deviceQuery), telling me devices are not available or don''t exist. I had to manually create the device files (/dev/nvidiactl, /dev/nvidia[0-9]) and while in runlevel 3 I had to create a persistent client with ''nvidia-smi -pm 1'' to avoid the initial driver state caching so that applications don''t have an initial slow-down when the CUDA runtime library opens the devices files for the first time. Have you been able to do so otherwise? I''m not sure what exactly you mean by "running the NVIDIA binary driver". However, installing the driver on Dom0/3.0.0-rc5 fails for me. The build process executes a configuration test (conftest.sh:1488 after extracting the driver sources) to determine the which Makefile to use, then fails. conftest.sh:1517 in <extracted nvidia sources>/kernel/ removing the stdout/stderr redirection: <src>/include/linux/rcupdate.h: In function ''__kfree_rcu'': <src>/include/linux/rcupdate.h:822: error: size of array ''type name'' is negative -------------------------------- $ sh devdriver_4.0_linux_64_270.41.19.run -x $ cd NVIDIA-Linux-x86_64-270.41.19/kernel $ make module If you are using a Linux 2.4 kernel, please make sure you either have configured kernel sources matching your kernel or the correct set of kernel headers installed on your system. If you are using a Linux 2.6 kernel, please make sure you have configured kernel sources matching your kernel installed on your system. If you specified a separate output directory using either the "KBUILD_OUTPUT" or the "O" KBUILD parameter, make sure to specify this directory with the SYSOUT environment variable or with the equivalent nvidia-installer command line option. Depending on where and how the kernel sources (or the kernel headers) were installed, you may need to specify their location with the SYSSRC environment variable or the equivalent nvidia-installer command line option. *** Unable to determine the target kernel version. *** make: *** [select_makefile] Error 1 -------------------------------- This cannot be true as I just built it and booted into it. /lib/modules/$(uname -r)/build points to the correct sources. You did not encounter this hiccup?> >> gotten replies essentially saying they cannot provide assistance, and >> have been following the nvnews.com forums. However, now that I have >> IOMMU-capable processors (with VT-d) and a version of Xen which can >> successfully program this hardware, I am interested to determine if it >> is possible to program the IOMMU *for* Dom0 (i.e. the target is Dom0, >> not an HVM/pvops guest). >> >> I have successfully been able to launch an HVM guest, passing a few >> GPUs through. Installing an unmodified developer driver from NVIDIA >> works using the standard method (no SYSSRC=/-OUT= or anything) and >> CUDA applications execute as expected. Can this be done for Dom0, >> achieving the same result without requiring any involvement from >> NVIDIA or modifications to the driver? > > It should be no trouble.. albeit you might need to update the Linux > kernel to 2.6.39 or 3.0 to take advantage of the 1-1 P2M mapping code.I will look into this once the module can build properly...> >> >> My immediate interest is more to see if it "can be done" via a hack or >> something, not necessarily whether it would make sense for Xen to >> support this in the future. My goal with this email is to get feedback >> on two fronts: 1) is there a limitation due to the architecture, >> meaning that as Dom0 is pvops it cannot use VT-d, or is it a >> implementation addition to Xen that would be needed? 2) If the latter >> is true (programming needed), how much effort (code/time) would >> you/anyone estimate it would take to enable this to work, and could >> you provide some starting pointers for me to do so? I am untrained >> when it comes to the Xen sourcecode. >> >> I''m using Xen 4.1.1 and pv-ops linux (not upstream) 2.6.32.40 on an >> Intel X5660 with a Tylersburg chipset. My host OS is Fedora 13, but >> that needn''t be static. >> >> I originally asked this on xen-users, but was informed this mailing >> list would be better suited: >> http://lists.xensource.com/archives/html/xen-users/2011-06/msg00451.html >> >> Many thanks in advance! >> Alex >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> >> I''m looking to enable the NVIDIA CUDA driver/runtime stack to work in > >> Dom0 on Xen. I''ve contacted NVIDIA through various capacities and have > > > > I''ve been running the NVIDIA binary driver with 2.6.39 / 3.0 kernel. > > To avoid confusion here, with 2.6.39* I have been able to successfully..snip..> However, installing the driver on Dom0/3.0.0-rc5 fails for me. The > build process executes a configuration test (conftest.sh:1488 after > extracting the driver sources) to determine the which Makefile to use, > then fails. > > conftest.sh:1517 in <extracted nvidia sources>/kernel/ removing the > stdout/stderr redirection: > > <src>/include/linux/rcupdate.h: In function ''__kfree_rcu'': > <src>/include/linux/rcupdate.h:822: error: size of array ''type name'' is negative.. snip> *** Unable to determine the target kernel version. *** > > make: *** [select_makefile] Error 1 > -------------------------------- > > This cannot be true as I just built it and booted into it. > /lib/modules/$(uname -r)/build points to the correct sources. You did > not encounter this hiccup?Um, actually I did. I completly failed to notice it as 2.6.39 worked and under 3.0 the nouveau driver kicked in. Looks like NVIDIA needs to fix their script to deal with the 3.0 kernels. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel