R help - Sep 2003 - cannot allocate vector of size...

Hi,

I know that this question has been asked several times before but I haven't
been able to find an answer.

I'm running R on a P3 933 with 384 MB of RAM. The system is running Windows
XP Home. 

I'm trying to read a 1054 row by 30303 column matrix from a tab delimited
text file ("file.txt") into R using the following command:
   m <- matrix(scan(file ="file.txt"), nrow = 1054, ncol = 30303,
byrow TRUE)
When I use this command I get the following output:
   Read 31939362 items
   Error: cannot allocate vector of size 249526Kb

I have recently increased the amount of RAM in my system. Prior to this
I got only the 'Error...' message and not the 'Read ... items'
message.

I have increased the space available to R at startup to 1G by modifying the
startup properties.

I would love any assistance/ideas anyone can give me with this problem.

Thanks!

Hannah

Hi, it seems that you can do

 s <- scan(file ="file.txt")

but that you do not have memory left to do

  m <- matrix(s, nrow = 1054, ncol = 30303, byrow = TRUE)

which requires twice the amount of memory. Could this be the case? 

First, try to read your matrix using read.matrix() that you'll find in
library(tseries). Second, what data type does you file contain? Is it
real numbers or integers? If you have integers I believe you could save
some memory. Try for instance,

i <- 1:10
d <- as.double(i)
object.size(i) # gives 68 bytes
object.size(d) # gives 108 bytes

You can help scan() by specifying what data type you are using. See
argument 'what' in ?scan. Of course, if you have mixed column types and
30303 columns this might need some work.

Henrik Bengtsson
Lund University
> -----Original Message-----
> From: r-help-bounces at stat.math.ethz.ch 
> [mailto:r-help-bounces at stat.math.ethz.ch] On Behalf Of 
> hwood at iprimus.com.au
> Sent: den 8 september 2003 08:21
> To: r-help at stat.math.ethz.ch
> Subject: [R] cannot allocate vector of size...
> 
> 
> Hi,
> 
> I know that this question has been asked several times before 
> but I haven't been able to find an answer.
> 
> I'm running R on a P3 933 with 384 MB of RAM. The system is 
> running Windows XP Home. 
> 
> I'm trying to read a 1054 row by 30303 column matrix from a 
> tab delimited text file ("file.txt") into R using the 
> following command:
>    m <- matrix(scan(file ="file.txt"), nrow = 1054, ncol = 
> 30303, byrow > TRUE)
> When I use this command I get the following output:
>    Read 31939362 items
>    Error: cannot allocate vector of size 249526Kb
> 
> I have recently increased the amount of RAM in my system. 
> Prior to this I got only the 'Error...' message and not the 
> 'Read ... items' message.
> 
> I have increased the space available to R at startup to 1G by 
> modifying the startup properties.
> 
> I would love any assistance/ideas anyone can give me with 
> this problem.
> 
> Thanks!
> 
> Hannah
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list 
> https://www.stat.math.ethz.ch/mailman/listinfo> /r-help
> 
>

Dear Hannah,

My usual replies to this question are:
  1. Read in just a subset (or a few random subsets) of
     your data and do some analysis on each of these
  2. Write some C code (either standalone or as something)
     that can be called from R
  3. Get even more RAM


Regards,

Andrew C. Ward

CAPE Centre
Department of Chemical Engineering
The University of Queensland
Brisbane Qld 4072 Australia
andreww at cheque.uq.edu.au


Quoting hwood at iprimus.com.au:
> Hi,
> 
> I know that this question has been asked several times
> before but I haven't
> been able to find an answer.
> 
> I'm running R on a P3 933 with 384 MB of RAM. The system
> is running Windows
> XP Home. 
> 
> I'm trying to read a 1054 row by 30303 column matrix from
> a tab delimited
> text file ("file.txt") into R using the following
> command:
>    m <- matrix(scan(file ="file.txt"), nrow = 1054, ncol
> = 30303, byrow > TRUE)
> When I use this command I get the following output:
>    Read 31939362 items
>    Error: cannot allocate vector of size 249526Kb
> 
> I have recently increased the amount of RAM in my system.
> Prior to this
> I got only the 'Error...' message and not the 'Read ...
> items' message.
> 
> I have increased the space available to R at startup to
> 1G by modifying the
> startup properties.
> 
> I would love any assistance/ideas anyone can give me with
> this problem.
> 
> Thanks!
> 
> Hannah
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://www.stat.math.ethz.ch/mailman/listinfo/r-help
>