R help - Aug 2007 - Cleaning up the memory

Hi,
 
I have 4 huge tables on which i want to do a PCA analysis and a kmean
clustering. If i run each table individually i have no problems, but if i want
to run it in a for loop i exceed the memory alocation after the second table,
even if i save the results as a csv table and i clean up all the big objects
with rm command. To me it seems that even if i don't have the objects
anymore, the memory these objects used to occupy is not cleared. Is there any
way to clear up the memory as well? I don't want to close R and start it up
again. Also i am running R under Windows.
 
thanks,
 
Monica
_________________________________________________________________
[[trailing spam removed]]

	[[alternative HTML version deleted]]

On Fri, 10 Aug 2007, Monica Pisica wrote:
>
> Hi,
>
> I have 4 huge tables on which i want to do a PCA analysis and a kmean 
> clustering. If i run each table individually i have no problems, but if 
> i want to run it in a for loop i exceed the memory alocation after the 
> second table, even if i save the results as a csv table and i clean up 
> all the big objects with rm command. To me it seems that even if i
don't
> have the objects anymore, the memory these objects used to occupy is not 
> cleared. Is there any way to clear up the memory as well? I don't want 
> to close R and start it up again. Also i am running R under Windows.
See ?gc, which does the clearing.

However, unless you study the memory allocation in detail (which you 
cannot do from R code), you don't actually know that this is the problem. 
More likely is that you have fragmentation of your 32-bit address space: 
see ?"Memory-limits".

Without any idea what memory you have and what 'huge' means, we can only
make wild guesses.  It might be worth raising the memory limit (the 
--max-mem-size flag).
>
> thanks,
>
> Monica
> _________________________________________________________________
> [[trailing spam removed]]
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

Thanks! I will look into ...
 
I have 4 GB RAM, and i was monitoring the memory with Windows task manager so i
was looking how R "gets" more and more memory allocation from less
than 100Mb to .... 1500Mb .....
 
My initial tables are between 30 to 80 Mb and the resulting tables that
incorporate the initial tables plus PCA and kmeans results are inbetween 50 to
200MB or thereabouts!
 
And yes, i don't really care about memory allocation in detail - what i want
is to free that memory after every cycle ;-)
 
Although, after i didn't do anything in R and it was idle for more than 30
min. the memory allocation according to Task manager dropped to 15 Mb .....
which is good - but i cannot wait inbetween cycles half an hour though .....
 
Again thanks,
 
Monica> Date: Fri, 10 Aug 2007 18:28:07 +0100> From:
ripley@stats.ox.ac.uk> To: pisicandru@hotmail.com> CC:
r-help@stat.math.ethz.ch> Subject: Re: [R] Cleaning up the memory> > On
Fri, 10 Aug 2007, Monica Pisica wrote:> > >> > Hi,> >>
> I have 4 huge tables on which i want to do a PCA analysis and a kmean >
> clustering. If i run each table individually i have no problems, but if
> > i want to run it in a for loop i exceed the memory alocation after the
> > second table, even if i save the results as a csv table and i clean up
> > all the big objects with rm command. To me it seems that even if i
don't > > have the objects anymore, the memory these objects used to
occupy is not > > cleared. Is there any way to clear up the memory as
well? I don't want > > to close R and start it up again. Also i am
running R under Windows.> > See ?gc, which does the clearing.> >
However, unless you study the memory allocation in detail (which you > cannot
do from R code), you don't actually know that this is the problem. > More
likely is that you have fragmentation of your 32-bit address space: > see
?"Memory-limits".> > Without any idea what memory you have and
what 'huge' means, we can only > make wild guesses. It might be worth
raising the memory limit (the > --max-mem-size flag).> > >> >
thanks,> >> > Monica> >
_________________________________________________________________> >
[[trailing spam removed]]> >> > [[alternative HTML version
deleted]]> >> > ______________________________________________>
> R-help@stat.math.ethz.ch mailing list> >
https://stat.ethz.ch/mailman/listinfo/r-help> > PLEASE do read the posting
guide http://www.R-project.org/posting-guide.html> > and provide
commented, minimal, self-contained, reproducible code.> >> > -- >
Brian D. Ripley, ripley@stats.ox.ac.uk> Professor of Applied Statistics,
http://www.stats.ox.ac.uk/~ripley/> University of Oxford, Tel: +44 1865
272861 (self)> 1 South Parks Road, +44 1865 272866 (PA)> Oxford OX1 3TG,
UK Fax: +44 1865 272595
_________________________________________________________________
Messenger Café — open for fun 24/7. Hot games, cool activities served daily.
Visit now.

	[[alternative HTML version deleted]]