Dear All, I have large matrix (46000 x 11250). I would like to do the linear regression for each row. I wrote a simple function that has lm() and used apply(mat,1,func). The issue is that it takes ages to load the file and also to finish the lm. I am using LINUX 64 bit with 32G mem. Is there an elegant and fast way of completing this task? Thanks in advance. Kind regards, Ezhil
Hi, On Sep 11, 2009, at 1:15 PM, A Ezhil wrote:> Dear All, > > I have large matrix (46000 x 11250). I would like to do the linear > regression for each row. I wrote a simple function that has lm() and > used apply(mat,1,func). The issue is that it takes ages to load the > file and also to finish the lm. I am using LINUX 64 bit with 32G > mem. Is there an elegant and fast way of completing this task?Do bigmemory and biglm help? http://cran.r-project.org/web/packages/bigmemory http://cran.r-project.org/web/packages/biglm -steve -- Steve Lianoglou Graduate Student: Computational Systems Biology | Memorial Sloan-Kettering Cancer Center | Weill Medical College of Cornell University Contact Info: http://cbio.mskcc.org/~lianos/contact
HI Steve, Thanks a lot for your suggestions. I never used bigmemory and biglm before. Is there a way (using systems commands) to combine awk and R? I was thinking of calling a R program inside Awk, passing each row to R, do the lm, get back results and process it with awk. Will this make the calculation fast? If you have any experience with large matrices, could you please share it with me? Thanks again, Ezhil --- On Fri, 9/11/09, Steve Lianoglou <mailinglist.honeypot at gmail.com> wrote:> From: Steve Lianoglou <mailinglist.honeypot at gmail.com> > Subject: Re: [R] Working with large matrix > To: "A Ezhil" <ezhil02 at yahoo.com> > Cc: R-help at r-project.org > Date: Friday, September 11, 2009, 11:56 PM > Hi, > > On Sep 11, 2009, at 1:15 PM, A Ezhil wrote: > > > Dear All, > > > > I have large matrix (46000 x 11250). I would like to > do the linear regression for each row. I wrote a simple > function that has lm() and used apply(mat,1,func). The issue > is that it takes ages to load the file and also to finish > the lm. I am using LINUX 64 bit with 32G mem. Is there an > elegant and fast way of completing this task? > > Do bigmemory and biglm help? > > http://cran.r-project.org/web/packages/bigmemory > http://cran.r-project.org/web/packages/biglm > > -steve > > -- > Steve Lianoglou > Graduate Student: Computational Systems Biology > ? |? Memorial Sloan-Kettering Cancer Center > ? |? Weill Medical College of Cornell University > Contact Info: http://cbio.mskcc.org/~lianos/contact > >
On Fri, Sep 11, 2009 at 12:15 PM, A Ezhil <ezhil02 at yahoo.com> wrote:> Dear All,> I have large matrix (46000 x 11250). I would like to do the linear regression for each row. I wrote a simple function that has lm() and used apply(mat,1,func). The issue is that it takes ages to load the file and also to finish the lm. I am using LINUX 64 bit with 32G mem. Is there an elegant and fast way of completing this task?I'm not quite sure what you mean by "do the linear regression for each row" but you may find it convenient to use a matrix as the response in the call to lm. In that case lm fits the same model, described by the right hand side of the formula, to each column of the matrix on the left hand side.