Hi, I've collected quite a bit of elevation data (LIDAR elevation points) and am looking for a suitable platform to do analysis and modeling on it. The data is sitting in an Oracle database, one table, 200 million rows of x,y, and z. I'm trying to figure out what hardware resources we need to reserve in order to run 64 BIT R for this size data. Here's my question: Is the 64 BIT version of R appropriate for this size? Or is attempting to read all 200 million rows a pipe dream no matter what platform I'm using? I've read in the archives that some have gotten 64 BIT R to build "out of box" on Intel Xeon processors...is there any differences in compiling/performance between the Intel Xeon architecture vs AMD 64? Are there pure physical limitations of 64 BIT R that would make intalling it on a grid (http://www.ncsu.edu/itd/hpc/Hardware/Hardware.php) a wasted effort? We have the memory and drive space to do this. Thanks for any advice Tom Colson Center for Earth Observation North Carolina State University Raleigh, NC 27695 (919) 515 3434 (919) 673 8023 tom_colson at ncsu.edu Online Calendar: http://www4.ncsu.edu/~tpcolson
On Sun, 13 Feb 2005, Thomas Colson wrote:> Hi, > > I've collected quite a bit of elevation data (LIDAR elevation points) and am > looking for a suitable platform to do analysis and modeling on it. The data > is sitting in an Oracle database, one table, 200 million rows of x,y, and z. > I'm trying to figure out what hardware resources we need to reserve in order > to run 64 BIT R for this size data. > > Here's my question: > Is the 64 BIT version of R appropriate for this size? Or is attempting to > read all 200 million rows a pipe dream no matter what platform I'm using?In principle R can handle this with enough memory. However, 200 million rows and three columns is 4.8Gb of storage, and R usually needs a few times the size of the data for working space. You would likely be better off not reading the whole data set at once, but loading sections of it from Oracle as needed. -thomas