R devel - Nov 2020 - .Internal(quit(...)): system call failed: Cannot allocate memory

Dear R-developers,

Some of the more fat scripts (50+ GB mem used by R) that I am running,
when they finish they do quit with q("no", status=0)
Quite often it happens that there is an extra stderr output produced
at the very end which looks like this:

Warning message:
In .Internal(quit(save, status, runLast)) :
  system call failed: Cannot allocate memory

Is there any way to avoid this kind of warnings? I am using stderr
output for detecting failures in scripts and this warning is a false
positive of a failure.

Maybe quit function could wait little bit longer trying to allocate
before it raises this warning?

Best regards,
Jan Gorecki

On 21/11/2020 12:51 p.m., Jan Gorecki wrote:
> Dear R-developers,
> 
> Some of the more fat scripts (50+ GB mem used by R) that I am running,
> when they finish they do quit with q("no", status=0)
> Quite often it happens that there is an extra stderr output produced
> at the very end which looks like this:
> 
> Warning message:
> In .Internal(quit(save, status, runLast)) :
>    system call failed: Cannot allocate memory
> 
> Is there any way to avoid this kind of warnings? I am using stderr
> output for detecting failures in scripts and this warning is a false
> positive of a failure.
> 
> Maybe quit function could wait little bit longer trying to allocate
> before it raises this warning?
I don't know what waiting would accomplish.  Generally speaking the 
allocation functions in R will try garbage collection before failing, so 
it looks like you are in a situation where there really is no memory 
available.  (I think code can prevent gc; maybe your code is doing that 
and not re-enabling it?)

Having a reproducible example would help, but I imagine it's not easy to 
put one together.

Duncan Murdoch

On 11/21/20 6:51 PM, Jan Gorecki wrote:
> Dear R-developers,
>
> Some of the more fat scripts (50+ GB mem used by R) that I am running,
> when they finish they do quit with q("no", status=0)
> Quite often it happens that there is an extra stderr output produced
> at the very end which looks like this:
>
> Warning message:
> In .Internal(quit(save, status, runLast)) :
>    system call failed: Cannot allocate memory
>
> Is there any way to avoid this kind of warnings? I am using stderr
> output for detecting failures in scripts and this warning is a false
> positive of a failure.
>
> Maybe quit function could wait little bit longer trying to allocate
> before it raises this warning?
If you see this warning, some call to system() or system2() or similar, 
which executes an external program, failed to even run a shell to run 
that external program, because there was not enough memory. You should 
be able to find out where it happens by checking the exit status of 
system().

Tomas

>
> Best regards,
> Jan Gorecki
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

Try explicitly deleting large data objects by calling `rm`, then `gc`.


On Mon, Nov 23, 2020 at 6:15 AM Tomas Kalibera <tomas.kalibera at
gmail.com>
wrote:
> On 11/21/20 6:51 PM, Jan Gorecki wrote:
> > Dear R-developers,
> >
> > Some of the more fat scripts (50+ GB mem used by R) that I am running,
> > when they finish they do quit with q("no", status=0)
> > Quite often it happens that there is an extra stderr output produced
> > at the very end which looks like this:
> >
> > Warning message:
> > In .Internal(quit(save, status, runLast)) :
> >    system call failed: Cannot allocate memory
> >
> > Is there any way to avoid this kind of warnings? I am using stderr
> > output for detecting failures in scripts and this warning is a false
> > positive of a failure.
> >
> > Maybe quit function could wait little bit longer trying to allocate
> > before it raises this warning?
>
> If you see this warning, some call to system() or system2() or similar,
> which executes an external program, failed to even run a shell to run
> that external program, because there was not enough memory. You should
> be able to find out where it happens by checking the exit status of
> system().
>
> Tomas
>
>
> >
> > Best regards,
> > Jan Gorecki
> >
> > ______________________________________________
> > R-devel at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
-- 
"Whereas true religion and good morals are the only solid foundations of
public liberty and happiness . . . it is hereby earnestly recommended to
the several States to take the most effectual measures for the
encouragement thereof." Continental Congress, 1778

	[[alternative HTML version deleted]]

The call to system() probably is an internal call used to delete the
session's tempdir().  This sort of failure means that a potentially large
amount of disk space is not being recovered when R is done.  Perhaps
R_CleanTempDir() could call R_unlink() instead of having a subprocess call
'rm -rf ...'.  Then it could also issue a specific warning if it was
impossible to delete all of tempdir().  (That should be very rare.)
> q("no")
Breakpoint 1, R_system (command=command at entry=0x7fffffffa1e0 "rm -Rf
/tmp/RtmppoKPXb") at sysutils.c:311
311     {
(gdb) where
#0  R_system (command=command at entry=0x7fffffffa1e0 "rm -Rf
/tmp/RtmppoKPXb") at sysutils.c:311
#1  0x00005555557c30ec in R_CleanTempDir () at sys-std.c:1178
#2  0x00005555557c31d7 in Rstd_CleanUp (saveact=<optimized out>, status=0,
runLast=<optimized out>) at sys-std.c:1243
#3  0x00005555557c593d in R_CleanUp (saveact=saveact at entry=SA_NOSAVE,
status=status at entry=0, runLast=<optimized out>) at system.c:87
#4  0x00005555556cc85e in do_quit (call=<optimized out>, op=<optimized
out>, args=0x555557813f90, rho=<optimized out>) at main.c:1393

-Bill

On Mon, Nov 23, 2020 at 3:15 AM Tomas Kalibera <tomas.kalibera at
gmail.com>
wrote:
> On 11/21/20 6:51 PM, Jan Gorecki wrote:
> > Dear R-developers,
> >
> > Some of the more fat scripts (50+ GB mem used by R) that I am running,
> > when they finish they do quit with q("no", status=0)
> > Quite often it happens that there is an extra stderr output produced
> > at the very end which looks like this:
> >
> > Warning message:
> > In .Internal(quit(save, status, runLast)) :
> >    system call failed: Cannot allocate memory
> >
> > Is there any way to avoid this kind of warnings? I am using stderr
> > output for detecting failures in scripts and this warning is a false
> > positive of a failure.
> >
> > Maybe quit function could wait little bit longer trying to allocate
> > before it raises this warning?
>
> If you see this warning, some call to system() or system2() or similar,
> which executes an external program, failed to even run a shell to run
> that external program, because there was not enough memory. You should
> be able to find out where it happens by checking the exit status of
> system().
>
> Tomas
>
>
> >
> > Best regards,
> > Jan Gorecki
> >
> > ______________________________________________
> > R-devel at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
	[[alternative HTML version deleted]]