R help - Nov 2005 - R seems to "stall" after several hours on a long series o f analyses... where to start?

You can test if the problem is accumulation in memory registers, which is
certainly what this sounds like. Just do a loop over a reasonably small
number of iterations and store or print the time between each iteration. If
memory accumulation it will run optimally for the first few iterations,
after which the time will increase noticeably (essentially exponentially,
hence ultimately freezes up). If this is the problem you may need to switch
to a For loop approach, or effect the loop as a DOS script to do each
iteration as a batch job (open/close R each iteration, old bytes can't
clutter the memory).

>	Mark Fowler
		Population Ecology Division
>	Bedford Inst of Oceanography
>	Dept Fisheries & Oceans
>	Dartmouth NS Canada
		B2Y 4A2
		Tel. (902) 426-3529
		Fax (902) 426-9710
		Email fowlerm at mar.dfo-mpo.gc.ca
		Home Tel. (902) 461-0708
		Home Email mark.fowler at ns.sympatico.ca


-----Original Message-----
From: David L. Van Brunt, Ph.D. [mailto:dlvanbrunt at gmail.com] 
Sent: November 7, 2005 7:16 AM
To: r-help at stat.math.ethz.ch
Subject: [R] R seems to "stall" after several hours on a long series
of
analyses... where to start?

Not sure where to even start on this.... I'm hoping there's some
debugging I
can do...

I have a loop that cycles through several different data sets (same
structure, different info), performing randomForest growth and
predictions... saving out the predictions for later study...

I get about 5 hours in (9%... of the planned iterations.. yikes!) and R just
freezes.

This happens in interactive and batch mode execution (I can see from the
".Rout" file that it gets about 9% through in Batch mode, and about 6%
if in
interactive mode... does that suggest memory problems?)

I'm thinking of re-executing this same code on a different platform to see
if that's the issue (currently using OS X)... any other suggestions on where
to look, or what to try to get more information?

Sorry so vague... it's a LOT of code, runs fine without error for many
iterations, so I didn't think the problem was syntax...

--
---------------------------------------
David L. Van Brunt, Ph.D.
mailto:dlvanbrunt at gmail.com

	[[alternative HTML version deleted]]

______________________________________________
R-help at stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html

On Mon, 7 Nov 2005, Fowler, Mark wrote:
> You can test if the problem is accumulation in memory registers, which is
> certainly what this sounds like. Just do a loop over a reasonably small
> number of iterations and store or print the time between each iteration.
If you are worried about memory accumulation it would make more sense to 
print memory use statistics after each iteration rather than the time (eg 
gc(,reset=TRUE))

 	-thomas


> If
> memory accumulation it will run optimally for the first few iterations,
> after which the time will increase noticeably (essentially exponentially,
> hence ultimately freezes up). If this is the problem you may need to switch
> to a For loop approach, or effect the loop as a DOS script to do each
> iteration as a batch job (open/close R each iteration, old bytes can't
> clutter the memory).
>
>
>> 	Mark Fowler
> 		Population Ecology Division
>> 	Bedford Inst of Oceanography
>> 	Dept Fisheries & Oceans
>> 	Dartmouth NS Canada
> 		B2Y 4A2
> 		Tel. (902) 426-3529
> 		Fax (902) 426-9710
> 		Email fowlerm at mar.dfo-mpo.gc.ca
> 		Home Tel. (902) 461-0708
> 		Home Email mark.fowler at ns.sympatico.ca
>
>
> -----Original Message-----
> From: David L. Van Brunt, Ph.D. [mailto:dlvanbrunt at gmail.com]
> Sent: November 7, 2005 7:16 AM
> To: r-help at stat.math.ethz.ch
> Subject: [R] R seems to "stall" after several hours on a long
series of
> analyses... where to start?
>
> Not sure where to even start on this.... I'm hoping there's some
debugging I
> can do...
>
> I have a loop that cycles through several different data sets (same
> structure, different info), performing randomForest growth and
> predictions... saving out the predictions for later study...
>
> I get about 5 hours in (9%... of the planned iterations.. yikes!) and R
just
> freezes.
>
> This happens in interactive and batch mode execution (I can see from the
> ".Rout" file that it gets about 9% through in Batch mode, and
about 6% if in
> interactive mode... does that suggest memory problems?)
>
> I'm thinking of re-executing this same code on a different platform to
see
> if that's the issue (currently using OS X)... any other suggestions on
where
> to look, or what to try to get more information?
>
> Sorry so vague... it's a LOT of code, runs fine without error for many
> iterations, so I didn't think the problem was syntax...
>
> --
> ---------------------------------------
> David L. Van Brunt, Ph.D.
> mailto:dlvanbrunt at gmail.com
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html
>
Thomas Lumley			Assoc. Professor, Biostatistics
tlumley at u.washington.edu	University of Washington, Seattle