R devel - Feb 2007 - memory problem read.table v array (PR#9526)

Full_Name: bill langdon
Version: 2.4.1
OS: ubuntu
Submission from: (NULL) (155.245.58.159)


#WBL 22 Feb 2007 ubuntu 
R.version
#platform       i486-pc-linux-gnu
#arch           i486
#os             linux-gnu
#system         i486, linux-gnu
#status
#major          2
#minor          4.1
#year           2006
#month          12
#day            18
#svn rev        40228
#language       R
#version.string R version 2.4.1 (2006-12-18)

#if matrix "a" is created by array "vals" is created ok

#if matrix "a" is created by read.table,
#peak resource use (CPU, memory) by "array()" is excessive

#a = array(0,dim=c(409600,1));                        #ok
#a = read.table("big.txt",header=FALSE);              #all memory used
#a = read.table("639x639.txt",header=FALSE);          #all memory used
#a = read.table("4096.txt",header=FALSE);             #all memory used
#a = read.table("4096nocomment.txt",header=FALSE);    #all memory used
#a = read.table("tiny.txt",header=FALSE);             #ten lines ok
#a = read.table("1000.txt",header=FALSE);             #all memory used
a = read.table("639.txt",header=FALSE);               #uses
1.6047029GB
dim(a)
dd = 639;
vals = array(a,dim=c(dd,dd));

There is nothing to reproduce here: we do not have 639.txt.

But note that read.table returns a data frame, and ?array has

     data: a vector (including a list) giving data to fill the array.

so I do wonder if this is what you intended: you seem to have tried to 
create an array list with 639*639 elements.  It is certainly not the same 
sort of object as array(0,dim=c(409600,1)).

Had 639.txt contained 639 rows of reals, your 'a' would be about
639*639*639*8 bytes, beyond the address space of your machine.

as.matrix(read.table("639.txt", header=FALSE)) might have been what
you
are looking for.


On Thu, 22 Feb 2007, wlangdon at essex.ac.uk wrote:
> Full_Name: bill langdon
> Version: 2.4.1
> OS: ubuntu
> Submission from: (NULL) (155.245.58.159)
>
>
> #WBL 22 Feb 2007 ubuntu
> R.version
> #platform       i486-pc-linux-gnu
> #arch           i486
> #os             linux-gnu
> #system         i486, linux-gnu
> #status
> #major          2
> #minor          4.1
> #year           2006
> #month          12
> #day            18
> #svn rev        40228
> #language       R
> #version.string R version 2.4.1 (2006-12-18)
>
> #if matrix "a" is created by array "vals" is created ok
>
> #if matrix "a" is created by read.table,
> #peak resource use (CPU, memory) by "array()" is excessive
>
> #a = array(0,dim=c(409600,1));                        #ok
> #a = read.table("big.txt",header=FALSE);              #all memory
used
> #a = read.table("639x639.txt",header=FALSE);          #all memory
used
> #a = read.table("4096.txt",header=FALSE);             #all memory
used
> #a = read.table("4096nocomment.txt",header=FALSE);    #all memory
used
> #a = read.table("tiny.txt",header=FALSE);             #ten lines
ok
> #a = read.table("1000.txt",header=FALSE);             #all memory
used
> a = read.table("639.txt",header=FALSE);               #uses
1.6047029GB
> dim(a)
> dd = 639;
> vals = array(a,dim=c(dd,dd));

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595