[R] sparse matrix, rnorm, malloc

g l
Sat, 10 Jun 2006 17:53:29 -0000

Hi,

I'm Sorry for any cross-posting. I've reviewed the archives and could
not find an exact answer to my question below.

I'm trying to generate very large sparse matrices (< 1% non-zero
entries per row). I have a sparse matrix function below which works
well until the row/col count exceeds 10,000. This is being run on a
machine with 32G memory:

sparse_matrix <- function(dims,rnd,p) {
         ptm <- proc.time()
         x <- round(rnorm(dims*dims),rnd)
         x[((abs(x) - p) < 0)] <- 0
         y <- matrix(x,nrow=dims,ncol=dims)
         proc.time() - ptm
}

When trying to generate the matrix around 20,000 rows/cols on a
machine with 32G of memory, the error message I receive is:

R(335) malloc: *** vm_allocate(size=3200004096) failed (error code=3)
R(335) malloc: *** error: can't allocate region
R(335) malloc: *** set a breakpoint in szone_error to debug
R(335) malloc: *** vm_allocate(size=3200004096) failed (error code=3)
R(335) malloc: *** error: can't allocate region
R(335) malloc: *** set a breakpoint in szone_error to debug
Error: cannot allocate vector of size 3125000 Kb
Error in round(rnorm(dims * dims), rnd) : unable to find the argument
'x' in selecting a method for function 'round'

* Last error line is obvious. Question:  on machine w/32G memory, why
can't it allocate a vector of size 3125000 Kb?

When trying to generate the matrix around 30,000 rows/cols, the error
message I receive is:

Error in rnorm(dims * dims) : cannot allocate vector of length 900000000
Error in round(rnorm(dims * dims), rnd) : unable to find the argument
'x' in selecting a method for function 'round'

* Last error line is obvious. Question: is this 900000000 bytes?
kilobytes? This error seems to be specific now to rnorm, but it
doesn't indicate the length metric (b/Kb/Mb) as it did for 20,000
rows/cols. Even if this Mb, why can't this be allocated on a machine
with 32G free memory?

When trying to generate the matrix with over 50,000 rows/cols, the
error message I receive is:

Error in rnorm(n, mean, sd) : invalid arguments
In addition: Warning message:
NAs introduced by coercion
Error in round(rnorm(dims * dims), rnd) : unable to find the argument
'x' in selecting a method for function 'round'

* Same.

Why would it generate different errors in each case? Code fixes? Any
simple ways to generate sparse matrices which would avoid above
problems?

Thanks in advance,

Gavin