R help - Dec 2008 - Clustering with Mahalanobis Distance

Dear R ExpeRts,

I'm having memory difficulties using mahalanobis distance to trying to
cluster in R.  I was wondering if anyone has done it with a matrix of 6525x17
(or something similar to that size).  I have a matrix of 6525 genes and 17
samples. I have my R memory increased to the max and am still getting
"cannot allocate vector of size" errors.  My matrix "x" is
actually a transpose of the original matrix (as I want to cluster by samples and
not genes). "y" is a vector of the mean gene expression levels and
"z" is the covariance matrix of "x" (I think this is where
the problem lies as the covariance matrix is enormous.

I can't really provide a reproducible example as I would have to attach my
data files, which I don't think anyone would appreciate.

rm(list=ls())  #removes everything from memory#
gc()                  #collects garbage#
memory.limit(size = 4095)   #increases memory limit#
x <- as.matrix(read.table("x.txt", header=TRUE, row.names=1))
y <- as.matrix(read.table("y.txt", header=TRUE, row.names=1))
z <- as.matrix(read.table("z.txt", header=TRUE, row.names=1))
mal <- mahalanobis(x, y, z)

The ultimate goal is to run "hclust" with the mahalanobis distance
matrix.


If anyone knows where I could find a more "memory friendly" function
or any advise as to what I might try to optimize my code,  I would appreciate
it.  sessionInfo() is below.

Many Thanks,

_____________________________________________
Patrick Richardson
Biostatistician - Program of Translational Medicine
Van Andel Research Institute - Webb Lab
333 Bostwick Avenue NE
Grand Rapids, MI  49503


R version 2.8.0 (2008-10-20)
i386-pc-mingw32

locale:
LC_COLLATE=English_United States.1252;LC_CTYPE=English_United
States.1252;LC_MONETARY=English_United
States.1252;LC_NUMERIC=C;LC_TIME=English_United States.1252

attached base packages:
[1] stats     graphics  grDevices datasets  tcltk     utils     methods   base

other attached packages:
[1] svSocket_0.9-5 svIO_0.9-5     R2HTML_1.59    svMisc_0.9-5   svIDE_0.9-5

loaded via a namespace (and not attached):
[1] tools_2.8.0

This email message, including any attachments, is for th...{{dropped:9}}

I don't have any experience with your particular problem, but the thing I
notice is that mahalanobis is that by default you specify a covariance
matrix, and it uses solve to calculate its inverse. If you could supply the
inverse covariance matrix (and specify inverted=TRUE to mahalanobis), that
might save a lot of memory.

If you cannot externally calculate the inverse before bringing it into R,
perhaps if you read only the covariance matrix and inverted it first, before
doing anything else? Or perhaps someone else knows some matrix magic?
-- 
View this message in context:
http://www.nabble.com/Clustering-with-Mahalanobis-Distance-tp20901487p20949816.html
Sent from the R help mailing list archive at Nabble.com.