Dear all, I am exploring ways to perform multiplication of a 90000 x 40000 matrix with it's transpose. As expected even a 40000 x 100 %*% 100x40000 didn't work on my desktop... giving the error "Error: cannot allocate vector of length 1600000000" However I am trying to run this on one node (64GB RAM; 2.60 GHz processor) of a high performance computing cluster. Appreciate if anyone has any comments on whether it's advisable to perform a matrix multiplication of this size using R and also on any better ways to handle this task. Kind Regards, Praveen. [[alternative HTML version deleted]]
Do your matrices have any special properties we should know about? For example, are they sparse, symmetric, diagonal, etc? -----Original Message----- From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On Behalf Of Praveen Surendran Sent: Wednesday, August 14, 2013 11:41 AM To: r-help at r-project.org Subject: [R] Matrix Multiplication using R. Dear all, I am exploring ways to perform multiplication of a 90000 x 40000 matrix with it's transpose. As expected even a 40000 x 100 %*% 100x40000 didn't work on my desktop... giving the error "Error: cannot allocate vector of length 1600000000" However I am trying to run this on one node (64GB RAM; 2.60 GHz processor) of a high performance computing cluster. Appreciate if anyone has any comments on whether it's advisable to perform a matrix multiplication of this size using R and also on any better ways to handle this task. Kind Regards, Praveen. [[alternative HTML version deleted]] ______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Well, one might start by noting that it's usually unnecessary and unwise to multiply a matrix by its transpose. Matrix decompositions, algebraic identities, and/or iterative procedures usually do calculations involving t(x)%*%x in better ways. 'Course without knowledge of your problem, maybe I'm wrong. -- Bert On Wed, Aug 14, 2013 at 8:40 AM, Praveen Surendran <ps629 at medschl.cam.ac.uk> wrote:> Dear all, > > I am exploring ways to perform multiplication of a 90000 x 40000 matrix with it's transpose. > As expected even a 40000 x 100 %*% 100x40000 didn't work on my desktop... giving the error "Error: cannot allocate vector of length 1600000000" > > However I am trying to run this on one node (64GB RAM; 2.60 GHz processor) of a high performance computing cluster. > Appreciate if anyone has any comments on whether it's advisable to perform a matrix multiplication of this size using R and also on any better ways to handle this task. > > Kind Regards, > > Praveen. > > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.-- Bert Gunter Genentech Nonclinical Biostatistics Internal Contact Info: Phone: 467-7374 Website: http://pharmadevelopment.roche.com/index/pdb/pdb-functional-groups/pdb-biostatistics/pdb-ncb-home.htm
In the event that these are moderately sparse matrices, you could try Matrix or SparseM. Roger Koenker rkoenker at illinois.edu On Aug 14, 2013, at 10:40 AM, Praveen Surendran wrote:> Dear all, > > I am exploring ways to perform multiplication of a 90000 x 40000 matrix with it's transpose. > As expected even a 40000 x 100 %*% 100x40000 didn't work on my desktop... giving the error "Error: cannot allocate vector of length 1600000000" > > However I am trying to run this on one node (64GB RAM; 2.60 GHz processor) of a high performance computing cluster. > Appreciate if anyone has any comments on whether it's advisable to perform a matrix multiplication of this size using R and also on any better ways to handle this task. > > Kind Regards, > > Praveen. > > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.