R devel - Mar 2005 - FW: R Process Killed out of logical swap space: Case id #259-4945 (PR#7732)

Dear Professor Ripley,

I am attempting a large scale mixed-model analysis of longitudinal gene
array data using R. I use a 64-bit SGI with 7GB available RAM and 40G
logical swap space together with unlimited virual swap space. The OS is IRIX
6.5.24. I use R version 1.9.0 since I have had problems upgrading to R 2.0.1
at 64-bit using native SGI compiler build.

I write a serial   for (i in 1:length(y)) { }
containing the call to lme as follows 

f1<-lme( expr ~ time + age + sex,random = ~ time | id,
correlation=cs3,data=gg.data.frame,method="ML",weights=varIdent(form=
~id))

After ~ 200 - 300 loops the process is killed by the OS with the following
errors

Mar 17 08:55:56 1A:helmholtz unix: ALERT: R.bin [1157] - out of logical swap
space during stack growth - see swap(1M) [filter /usr/sbin/klogpp failed:
killed by signal 9]
Mar 17 08:55:56 1A:helmholtz unix: ALERT: R.bin [1157] - out of logical swap
space during stack growth - see swap(1M)
Mar 17 08:55:56 1A:helmholtz unix: ALERT: R.bin [1157] - out of logical swap
space during stack growth - see swap(1M)
Mar 17 08:55:56 1A:helmholtz unix: ALERT: R.bin [1157] - out of logical swap
space during stack growth - see swap(1M)
Mar 17 08:55:56 1A:helmholtz unix: |$(0x6dc)ALERT: Process [R.bin] pid 1157
killed: not enough memory to grow stack
Mar 17 08:55:56 1A:helmholtz unix: |$(0x6dc)ALERT: Process [R.bin] pid 1157
killed: not enough memory to grow stack
Mar 17 08:55:56 1A:helmholtz unix: ALERT: syslogd[1642] was killed; Dynamic
Loader cannot get enough memory to map exe
Mar 17 08:55:56 1A:helmholtz unix: ALERT: syslogd[1642] was killed; Dynamic
Loader cannot get enough memory to map exe
Mar 17 08:55:56 1A:helmholtz unix: ALERT: syslogd [1642] - out of logical
swap space during exec - see swap(1M)
Mar 17 08:55:56 1A:helmholtz unix: ALERT: syslogd [1642] - out of logical
swap space during exec - see swap(1M)

I have observed the crash while examining the system memory and stack
allocations - these do not appear to be growing during execution of the
process loops. I have checked with SGI and the view is that this is likely
related to R. While there are several work-arounds it would be ideal to
'batch' this job and further it is unclear to me what is happening. I am
grateful for your attention and potential insights.

Sincerely

David F. Moore
Associate Professor of Vascular Neurology
University of Manitoba

This e-mail and/or any documents in this transmission is intended for the
address(s) only and may contain legally privileged or confidential information.
Any unauthorized use, disclosure, distribution, copying or dissemination is
strictly prohibited. If you receive this transmission in error, please notify
the sender immediately and return the original.

DMoore@exchange.hsc.mb.ca writes:
> Dear Professor Ripley,
Ummm, "PR" is for "Problem Report", not "Professor
Ripley"....
 
> I am attempting a large scale mixed-model analysis of longitudinal gene
> array data using R. I use a 64-bit SGI with 7GB available RAM and 40G
> logical swap space together with unlimited virual swap space. The OS is
IRIX
> 6.5.24. I use R version 1.9.0 since I have had problems upgrading to R
2.0.1
> at 64-bit using native SGI compiler build.
> 
> I write a serial   for (i in 1:length(y)) { }
> containing the call to lme as follows 
> 
> f1<-lme( expr ~ time + age + sex,random = ~ time | id,
>
correlation=cs3,data=gg.data.frame,method="ML",weights=varIdent(form=
~id))
> 
> After ~ 200 - 300 loops the process is killed by the OS with the following
> errors
> 
> Mar 17 08:55:56 1A:helmholtz unix: ALERT: R.bin [1157] - out of logical
swap
> space during stack growth - see swap(1M) [filter /usr/sbin/klogpp failed:
> killed by signal 9]
> Mar 17 08:55:56 1A:helmholtz unix: ALERT: R.bin [1157] - out of logical
swap
> space during stack growth - see swap(1M)
> Mar 17 08:55:56 1A:helmholtz unix: ALERT: R.bin [1157] - out of logical
swap
> space during stack growth - see swap(1M)
> Mar 17 08:55:56 1A:helmholtz unix: ALERT: R.bin [1157] - out of logical
swap
> space during stack growth - see swap(1M)
> Mar 17 08:55:56 1A:helmholtz unix: |$(0x6dc)ALERT: Process [R.bin] pid 1157
> killed: not enough memory to grow stack
> Mar 17 08:55:56 1A:helmholtz unix: |$(0x6dc)ALERT: Process [R.bin] pid 1157
> killed: not enough memory to grow stack
> Mar 17 08:55:56 1A:helmholtz unix: ALERT: syslogd[1642] was killed; Dynamic
> Loader cannot get enough memory to map exe
> Mar 17 08:55:56 1A:helmholtz unix: ALERT: syslogd[1642] was killed; Dynamic
> Loader cannot get enough memory to map exe
> Mar 17 08:55:56 1A:helmholtz unix: ALERT: syslogd [1642] - out of logical
> swap space during exec - see swap(1M)
> Mar 17 08:55:56 1A:helmholtz unix: ALERT: syslogd [1642] - out of logical
> swap space during exec - see swap(1M)
> 
> I have observed the crash while examining the system memory and stack
> allocations - these do not appear to be growing during execution of the
> process loops. I have checked with SGI and the view is that this is likely
> related to R. While there are several work-arounds it would be ideal to
> 'batch' this job and further it is unclear to me what is happening.
I am
> grateful for your attention and potential insights.
Hmm, I agree that this could well be a bug in R or the nlme package.
Sounds like a single huge memory allocation is attempted, which could
fairly easily happen as the result of a random memory overwrite.

However, it is not attractive to try and find bugs in old versions of
R (several bugs of this general variety have been fixed since 1.9.1),
and it is near impossible to find such bugs without a way of
reproducing the conditions. I.e. you are the one with the motivation
and probably also the only person with the means to pinpoint the bug. 

Can you run your code under a debugger?

-- 
   O__  ---- Peter Dalgaard             Blegdamsvej 3  
  c/ /'_ --- Dept. of Biostatistics     2200 Cph. N   
 (*) \(*) -- University of Copenhagen   Denmark      Ph: (+45) 35327918
~~~~~~~~~~ - (p.dalgaard@biostat.ku.dk)             FAX: (+45) 35327907