Hi all, I'm trying to read a 1220 * 20000 table in R but I'm having lot of problems. Basically what it happens is that R.bin starts eating all my memory until it gets about 90%. At that point it locks itself in a uninterruptible sleep status (at least that's what top says) where it just sits there barely using the cpu at all but keeping its tons of memory. I've tried with read.table and scan but none of them did the trick. I've also tried some orrible hack like reading one line a time and gradually combining everything in a matrix using rbind... nope! It seems I can read up to 500 lines in a *decent* time but nothing more. The machine is a 3 GHz P4 with HT and 512 MB RAM running R-1.8.1. Will I have to write a little a C program myself to handle this thing or am I missing something? Thanks in advance for your help, fede
Federico Gherardini wrote:> Hi all, > > I'm trying to read a 1220 * 20000 table in R but I'm having lot of problems. Basically what it happens is that R.bin starts eating all my memory until it gets about 90%. At that point it locks itself in a uninterruptible sleep status (at least that's what top says) where it just sits there barely using the cpu at all but keeping its tons of memory. I've tried with read.table and scan but none of them did the trick. I've also tried some orrible hack like reading one line a time and gradually combining everything in a matrix using rbind... nope! It seems I can read up to 500 lines in a *decent* time but nothing more. The machine is a 3 GHz P4 with HT and 512 MB RAM running R-1.8.1. Will I have to write a little a C program myself to handle this thing or am I missing something?If your data is numeric, you will need roughly 1220 * 20000 * 8 / 1024 / 1024 ~~ 200 MB just to store one copy in memory. If you need more than two copies, your machine with its 512MB will start to use swap space ..... Hence either use a machine with more memory, or don't use all the data at once in memory, e.g. by making use of a database. Uwe Ligges> Thanks in advance for your help, > > fede > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://www.stat.math.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
On Wed, 2004-07-28 at 03:10, Federico Gherardini wrote:> Hi all, > > I'm trying to read a 1220 * 20000 table in R but I'm having lot of problems. Basically what it happens is that R.bin starts eating all my memory until it gets about 90%. At that point it locks itself in a uninterruptible sleep status (at least that's what top says) where it just sits there barely using the cpu at all but keeping its tons of memory. I've tried with read.table and scan but none of them did the trick. I've also tried some orrible hack like reading one line a time and gradually combining everything in a matrix using rbind... nope! It seems I can read up to 500 lines in a *decent* time but nothing more. The machine is a 3 GHz P4 with HT and 512 MB RAM running R-1.8.1. Will I have to write a little a C program myself to handle this thing or am I missing something? > > Thanks in advance for your help, > > fede >Hi, It looks like you're running linux !? if so it will be quite easy to create a table in MySQL, upload all the data into the database and access the data with RMySQL (it's _very_ fast). Probably there will be some operations that you can do on MySQL instead of "eating" memory in R. Regards EJ
Hi, i'm working with a ~ 250.000 * 150 data.frame and can share your problems - i've upgraded last weekend my notebook from 512MB -> 1024MB, it's really better especially for load, write.table , mysqlReadTable, mysqlWriteTable, because machine begin caching if RAM is full. One example: With 512MB i get after some hours no success write a table to mysql. With 1024MB it does in some minutes. regards, christian Am Mittwoch, 28. Juli 2004 04:10 schrieb Federico Gherardini:> Hi all, > > I'm trying to read a 1220 * 20000 table in R but I'm having lot of > problems. Basically what it happens is that R.bin starts eating all my > memory until it gets about 90%. At that point it locks itself in a > uninterruptible sleep status (at least that's what top says) where it just > sits there barely using the cpu at all but keeping its tons of memory. I've > tried with read.table and scan but none of them did the trick. I've also > tried some orrible hack like reading one line a time and gradually > combining everything in a matrix using rbind... nope! It seems I can read > up to 500 lines in a *decent* time but nothing more. The machine is a 3 GHz > P4 with HT and 512 MB RAM running R-1.8.1. Will I have to write a little a > C program myself to handle this thing or am I missing something? > > Thanks in advance for your help, > > fede > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://www.stat.math.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide! > http://www.R-project.org/posting-guide.html