thr3ads.net - R help - [R] can we manage memory usage to increase speed? [Aug 2005]

If this information is useful, please help other people find it:
Share via:

Zhilin Liu

2005-Aug-02 00:27 UTC

[R] can we manage memory usage to increase speed?

Hi,
 
Thanks for reading.
 
I am running  a process in R for microarray data analysis. RedHat Enterprise
Linux 4, dual AMD CPU, 6G memory. However, the R process use only a total of
<200M memory. And the CPU usage is total to ~110% for two. The program takes
at least 2 weeks to run at the current speed. Is there some way we can increase
the usage of CPUs and memories and speed up? Any suggestion is appreciated.
 
Thanks again.
 
Zhilin 

	[[alternative HTML version deleted]]

Haibo Huang

2005-Aug-02 02:59 UTC

head link

[R] can we manage memory usage to increase speed?

Please refer to the following post.

Ed

--- Mike Lawrence <Mike.Lawrence at dal.ca> wrote:
> Date: Mon,  1 Aug 2005 00:19:06 -0300
> From: Mike Lawrence <Mike.Lawrence at dal.ca>
> To: "Briggs, Meredith M"
> <Meredith.Briggs at team.telstra.com>
> CC: r-help at stat.math.ethz.ch
> Subject: Re: [R] How do you increase memeory?
> 
> memory.limit(size = x)
> 
> where x is the desired memory limit in MB.
> 
> 
> 
> Quoting "Briggs, Meredith M"
> <Meredith.Briggs at team.telstra.com>:
> 
> > Hello
> >
> >
> > Function memory.size() =435109888. How do I
> increase it by, say 30%?
> >
> > Thanks
> > Meredith
> >
> > 	[[alternative HTML version deleted]]
> >
> > ______________________________________________
> > R-help at stat.math.ethz.ch mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
> >
> 
> 
> 
> -- 
> 
> Mike Lawrence, BA(Hons)
> Research Assistant to Dr. Gail Eskes
> Dalhousie University & QEII Health Sciences Centre
> (Psychiatry)
> 
> Mike.Lawrence at Dal.Ca
> 
> "The road to Wisdom? Well, it's plain and simple to
> express:
> Err and err and err again, but less and less and
> less."
> - Piet Hein
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
> 

--- Zhilin Liu <zhilinliu_email at yahoo.com> wrote:
> Hi,
>  
> Thanks for reading.
>  
> I am running  a process in R for microarray data
> analysis. RedHat Enterprise Linux 4, dual AMD CPU,
> 6G memory. However, the R process use only a total
> of <200M memory. And the CPU usage is total to ~110%
> for two. The program takes at least 2 weeks to run
> at the current speed. Is there some way we can
> increase the usage of CPUs and memories and speed
> up? Any suggestion is appreciated.
>  
> Thanks again.
>  
> Zhilin 
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
>

Prof Brian Ripley

2005-Aug-02 06:42 UTC

head link

[R] can we manage memory usage to increase speed?

On Mon, 1 Aug 2005, Haibo Huang wrote:
> Please refer to the following post.
Which is about Windows only, not Linux.  (And on Windows, the answer given 
is on the help page for memory.size. together with a better one.)
>
> Ed
>
> --- Mike Lawrence <Mike.Lawrence at dal.ca> wrote:
>
>> Date: Mon,  1 Aug 2005 00:19:06 -0300
>> From: Mike Lawrence <Mike.Lawrence at dal.ca>
>> To: "Briggs, Meredith M"
>> <Meredith.Briggs at team.telstra.com>
>> CC: r-help at stat.math.ethz.ch
>> Subject: Re: [R] How do you increase memeory?
>>
>> memory.limit(size = x)
>>
>> where x is the desired memory limit in MB.
>>
>>
>>
>> Quoting "Briggs, Meredith M"
>> <Meredith.Briggs at team.telstra.com>:
>>
>>> Hello
>>>
>>>
>>> Function memory.size() =435109888. How do I
>> increase it by, say 30%?
>>>
>>> Thanks
>>> Meredith
> --- Zhilin Liu <zhilinliu_email at yahoo.com> wrote:
>
>> Hi,
>>
>> Thanks for reading.
>>
>> I am running  a process in R for microarray data
>> analysis. RedHat Enterprise Linux 4, dual AMD CPU,
>> 6G memory. However, the R process use only a total
>> of <200M memory. And the CPU usage is total to ~110%
>> for two. The program takes at least 2 weeks to run
>> at the current speed. Is there some way we can
>> increase the usage of CPUs and memories and speed
>> up? Any suggestion is appreciated.
>>
>> Thanks again.
>>
>> Zhilin
>>
>> 	[[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-help at stat.math.ethz.ch mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide!
>> http://www.R-project.org/posting-guide.html
>>
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html
>
-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

Tuszynski, Jaroslaw W.

2005-Aug-02 13:16 UTC

head link

[R] can we manage memory usage to increase speed?

If you have a code that takes 2 weeks to run, than it might be a case of
inefficient algorithm design. I was able to go from overnight runs (SELDI
data analysis) to 20 minute runs by identifying single inefficient function
that took most of the time, and writing it in C.

Jarek
====================================================\======
 Jarek Tuszynski, PhD.                           o / \ 
 Science Applications International Corporation  <\__,|  
 (703) 676-4192                                   ">   \
 Jaroslaw.W.Tuszynski at saic.com                     `    \


-----Original Message-----
From: r-help-bounces at stat.math.ethz.ch
[mailto:r-help-bounces at stat.math.ethz.ch] On Behalf Of Zhilin Liu
Sent: Monday, August 01, 2005 8:28 PM
To: r-help at stat.math.ethz.ch
Subject: [R] can we manage memory usage to increase speed?

Hi,
 
Thanks for reading.
 
I am running  a process in R for microarray data analysis. RedHat Enterprise
Linux 4, dual AMD CPU, 6G memory. However, the R process use only a total of
<200M memory. And the CPU usage is total to ~110% for two. The program takes
at least 2 weeks to run at the current speed. Is there some way we can
increase the usage of CPUs and memories and speed up? Any suggestion is
appreciated.
 
Thanks again.
 
Zhilin 

	[[alternative HTML version deleted]]

______________________________________________
R-help at stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html

Spencer Graves

2005-Aug-02 15:51 UTC

head link

[R] can we manage memory usage to increase speed?

And you can identify inefficient code fairly easily taking snapshots 
from "proc.time" and computing elapsed time for sections of your code.

	  spencer graves

Tuszynski, Jaroslaw W. wrote:
>  
> If you have a code that takes 2 weeks to run, than it might be a case of
> inefficient algorithm design. I was able to go from overnight runs (SELDI
> data analysis) to 20 minute runs by identifying single inefficient function
> that took most of the time, and writing it in C.
> 
> Jarek
> ====================================================\======> 
>  Jarek Tuszynski, PhD.                           o / \ 
>  Science Applications International Corporation  <\__,|  
>  (703) 676-4192                                   ">   \
>  Jaroslaw.W.Tuszynski at saic.com                     `    \
> 
> 
> -----Original Message-----
> From: r-help-bounces at stat.math.ethz.ch
> [mailto:r-help-bounces at stat.math.ethz.ch] On Behalf Of Zhilin Liu
> Sent: Monday, August 01, 2005 8:28 PM
> To: r-help at stat.math.ethz.ch
> Subject: [R] can we manage memory usage to increase speed?
> 
> Hi,
>  
> Thanks for reading.
>  
> I am running  a process in R for microarray data analysis. RedHat
Enterprise
> Linux 4, dual AMD CPU, 6G memory. However, the R process use only a total
of
> <200M memory. And the CPU usage is total to ~110% for two. The program
takes
> at least 2 weeks to run at the current speed. Is there some way we can
> increase the usage of CPUs and memories and speed up? Any suggestion is
> appreciated.
>  
> Thanks again.
>  
> Zhilin 
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html
-- 
Spencer Graves, PhD
Senior Development Engineer
PDF Solutions, Inc.
333 West San Carlos Street Suite 700
San Jose, CA 95110, USA

spencer.graves at pdf.com
www.pdf.com <http://www.pdf.com>
Tel:  408-938-4420
Fax: 408-280-7915

Douglas Bates

2005-Aug-02 16:57 UTC

head link

[R] can we manage memory usage to increase speed?

On 8/2/05, Spencer Graves <spencer.graves at pdf.com>
wrote:>           And you can identify inefficient code fairly easily taking
snapshots
> from "proc.time" and computing elapsed time for sections of your
code.
> 
>           spencer graves
Using Rprof may be a better choice.  See

?Rprof
> 
> Tuszynski, Jaroslaw W. wrote:
> 
> >
> > If you have a code that takes 2 weeks to run, than it might be a case
of
> > inefficient algorithm design. I was able to go from overnight runs
(SELDI
> > data analysis) to 20 minute runs by identifying single inefficient
function
> > that took most of the time, and writing it in C.
> >
> > Jarek
> > ====================================================\======> >
> >  Jarek Tuszynski, PhD.                           o / \
> >  Science Applications International Corporation  <\__,|
> >  (703) 676-4192                                   ">   \
> >  Jaroslaw.W.Tuszynski at saic.com                     `    \
> >
> >
> > -----Original Message-----
> > From: r-help-bounces at stat.math.ethz.ch
> > [mailto:r-help-bounces at stat.math.ethz.ch] On Behalf Of Zhilin Liu
> > Sent: Monday, August 01, 2005 8:28 PM
> > To: r-help at stat.math.ethz.ch
> > Subject: [R] can we manage memory usage to increase speed?
> >
> > Hi,
> >
> > Thanks for reading.
> >
> > I am running  a process in R for microarray data analysis. RedHat
Enterprise
> > Linux 4, dual AMD CPU, 6G memory. However, the R process use only a
total of
> > <200M memory. And the CPU usage is total to ~110% for two. The
program takes
> > at least 2 weeks to run at the current speed. Is there some way we can
> > increase the usage of CPUs and memories and speed up? Any suggestion
is
> > appreciated.
> >
> > Thanks again.
> >
> > Zhilin
> >
> >       [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > R-help at stat.math.ethz.ch mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide!
> > http://www.R-project.org/posting-guide.html
> >
> > ______________________________________________
> > R-help at stat.math.ethz.ch mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html
> 
> --
> Spencer Graves, PhD
> Senior Development Engineer
> PDF Solutions, Inc.
> 333 West San Carlos Street Suite 700
> San Jose, CA 95110, USA
> 
> spencer.graves at pdf.com
> www.pdf.com <http://www.pdf.com>
> Tel:  408-938-4420
> Fax: 408-280-7915
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html
>

Zhilin Liu

2005-Aug-02 21:35 UTC

head link

[R] can we manage memory usage to increase speed?

Hi, 
Thank you all for the kind reply.

I recompiled R as the previous one turned profiling
off. 

I am using package MAANOVA, running the matest
function which is a permutation test. The author did
warn that it takes a long time to run.

Here is one of the test results:
[ Rdata]# ./R CMD Rprof maanovatest.out

Each sample represents 0.02 seconds.
Total run time: 905.959999999466 seconds.

Total seconds: time spent in function and callees.
Self seconds: time spent in function alone.

   %       total       %       self
 total    seconds     self    seconds    name
100.00    905.94      0.00      0.00     "matest"
 80.18    726.40      0.25      2.30     "fitmaanova"
 79.37    719.04      0.19      1.72     "mixed"
 68.34    619.16      1.05      9.50     "pinv"
 64.33    582.78      0.88      7.96     "La.svd"
 55.51    502.90     55.51    502.90     ".Call"
 38.47    348.54      1.58     14.28     "makeHq"
 34.50    312.60      0.13      1.18     "solveMME"
 19.80    179.42      0.18      1.60    
"matest.engine"
 10.19     92.30     10.19     92.30     "%*%"
......

The other part are not pasted as they are almost the
same everytime we check the profiling. Only the parts
above changes. For example, here is another output:
100.00   1411.02      0.00      0.02     "matest"
 82.88   1169.40      0.24      3.40     "fitmaanova"
 82.22   1160.18      0.19      2.74     "mixed"
 68.82    971.02      1.06     14.90     "pinv"
 64.84    914.94      0.85     11.98     "La.svd"
 56.10    791.64     56.10    791.64     ".Call"
 39.13    552.10      1.55     21.84     "makeHq"
 36.77    518.82      0.13      1.88     "solveMME"
 31.40    443.04      0.00      0.00     "matest.perm"
 17.10    241.32      0.16      2.28    
"matest.engine"
 10.15    143.24     10.15    143.24     "%*%"

I run this with a permutation of 2 times and it is
still running. So it is not possible to run 1000
permutations with this kind of speed.

And here is the output of TOP for R:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM   
TIME+  COMMAND
15250 liuz       0 -20  218m 136m 2556 R 72.3  2.5 
29:05.20 R

Any suggestion to improve the performance is highly
appreciated.

Thanks a lot.

Zhilin



--- Douglas Bates <dmbates at gmail.com> wrote:
> On 8/2/05, Spencer Graves <spencer.graves at pdf.com>
> wrote:
> >           And you can identify inefficient code
> fairly easily taking snapshots
> > from "proc.time" and computing elapsed time for
> sections of your code.
> > 
> >           spencer graves
> 
> Using Rprof may be a better choice.  See
> 
> ?Rprof
> 
> > 
> > Tuszynski, Jaroslaw W. wrote:
> > 
> > >
> > > If you have a code that takes 2 weeks to run,
> than it might be a case of
> > > inefficient algorithm design. I was able to go
> from overnight runs (SELDI
> > > data analysis) to 20 minute runs by identifying
> single inefficient function
> > > that took most of the time, and writing it in C.
> > >
> > > Jarek
> > >
>====================================================\======> >
>> > >  Jarek Tuszynski, PhD.                          
> o / \
> > >  Science Applications International Corporation 
> <\__,|
> > >  (703) 676-4192                                 
>  ">   \
> > >  Jaroslaw.W.Tuszynski at saic.com                  
>   `    \
> > >
> > >
> > > -----Original Message-----
> > > From: r-help-bounces at stat.math.ethz.ch
> > > [mailto:r-help-bounces at stat.math.ethz.ch] On
> Behalf Of Zhilin Liu
> > > Sent: Monday, August 01, 2005 8:28 PM
> > > To: r-help at stat.math.ethz.ch
> > > Subject: [R] can we manage memory usage to
> increase speed?
> > >
> > > Hi,
> > >
> > > Thanks for reading.
> > >
> > > I am running  a process in R for microarray data
> analysis. RedHat Enterprise
> > > Linux 4, dual AMD CPU, 6G memory. However, the R
> process use only a total of
> > > <200M memory. And the CPU usage is total to
> ~110% for two. The program takes
> > > at least 2 weeks to run at the current speed. Is
> there some way we can
> > > increase the usage of CPUs and memories and
> speed up? Any suggestion is
> > > appreciated.
> > >
> > > Thanks again.
> > >
> > > Zhilin
> > >
> > >       [[alternative HTML version deleted]]
> > >
> > > ______________________________________________
> > > R-help at stat.math.ethz.ch mailing list
> > > https://stat.ethz.ch/mailman/listinfo/r-help
> > > PLEASE do read the posting guide!
> > > http://www.R-project.org/posting-guide.html
> > >
> > > ______________________________________________
> > > R-help at stat.math.ethz.ch mailing list
> > > https://stat.ethz.ch/mailman/listinfo/r-help
> > > PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
> > 
> > --
> > Spencer Graves, PhD
> > Senior Development Engineer
> > PDF Solutions, Inc.
> > 333 West San Carlos Street Suite 700
> > San Jose, CA 95110, USA
> > 
> > spencer.graves at pdf.com
> > www.pdf.com <http://www.pdf.com>
> > Tel:  408-938-4420
> > Fax: 408-280-7915
> > 
> > ______________________________________________
> > R-help at stat.math.ethz.ch mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
> >
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
>

Possibly Parallel Threads

Search for more maybe matching threads

R help - Aug 2005 - can we manage memory usage to increase speed?

[R] can we manage memory usage to increase speed?

[R] can we manage memory usage to increase speed?

[R] can we manage memory usage to increase speed?

[R] can we manage memory usage to increase speed?

[R] can we manage memory usage to increase speed?

[R] can we manage memory usage to increase speed?

[R] can we manage memory usage to increase speed?

Possibly Parallel Threads