R help - Jun 2003 - data.frame building

Hi all,
I have two seperate questions that both deal with the way R handels data
sets:

First, I am trying to read a data set of 80M into R. I am using
read.table(). The file is a tab file and I have tested the function for a
small amount of lines. It seems to work fine (i.e. correct amount of columns
and rows). However, when I try to read the full data set, R outputs a memory
limit error and stops responding to any command even the gc()  and the
quit() commands. It says it reached maximum memory of 489MB (that is the
amount of RAM I have). I tried the to increase vsize and nsize but no luck.

My second question concerns building data frames in a loop. I know a while
back Prof. Ripley has suggested to construct a data frame of the right size
outside the loop and fill it within the loop. unfortunately, it is taken so
long (this is on P4 2.4 G 489M RAM). I am wondering if there was any other
way that is faster to do this job. I also do not jnow why it is faster to
construct columns in a data.frame than it is to do rows. i.e. I had to
construct 719 columns and it does not take any time to do so (each) however
if I say data[i,]<-data2[j,] it takes so long versus
data$temp<-data$a==data$b for example.


any help is greatly appreciated

Jean Eid

On Mon, 2 Jun 2003, Jean Eid wrote:
> I have two seperate questions that both deal with the way R handels data
> sets:
> 
> First, I am trying to read a data set of 80M into R. I am using
> read.table(). The file is a tab file and I have tested the function for a
> small amount of lines. It seems to work fine (i.e. correct amount of
columns
> and rows). However, when I try to read the full data set, R outputs a
memory
> limit error and stops responding to any command even the gc()  and the
> quit() commands. It says it reached maximum memory of 489MB (that is the
> amount of RAM I have). I tried the to increase vsize and nsize but no luck.
Have you read and used the hints in help(read.table) and the R Data
Import/Export Manual?  Alternatively, use scan() (as hinted there).

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595