R devel - May 2016 - R process killed when allocating too large matrix (Mac OS X)

On 05/05/2016 10:11, Uwe Ligges wrote:
> Actually this also happens under Linux and I had my R processes killed
> more than once (and much worse also other processes so that we had to
> reboot a server, essentially).
I found that setting RLIMIT_AS [1] works very well on Linux. But this
requires that you cap memory to some fixed value.
> library(RAppArmor)
> rlimit_as(1e9)
> rnorm(1e9)
Error: cannot allocate vector of size 7.5 Gb

The RAppArmor package has many other utilities to protect your server
such from a mis-behaving process such as limiting cpu time
(RLIMIT_CPU), fork bombs (RLIMIT_NPROC) and file sizes (RLIMIT_FSIZE).

[1] http://linux.die.net/man/2/getrlimit

My ulimit package exposes this API ([1], should finally submit it to 
CRAN); unfortunately this very API seems to be unsupported on OS X 
[2,3]. Last time I looked into it, neither of the documented settings 
achieved the desired effect.


-Kirill


[1] http://krlmlr.github.io/ulimit
[2] 
http://stackoverflow.com/questions/3274385/how-to-limit-memory-of-a-os-x-program-ulimit-v-neither-m-are-working
[3] 
https://developer.apple.com/library/ios/documentation/System/Conceptual/ManPages_iPhoneOS/man2/getrlimit.2.html


On 10.05.2016 01:08, Jeroen Ooms wrote:
> On 05/05/2016 10:11, Uwe Ligges wrote:
>> Actually this also happens under Linux and I had my R processes killed
>> more than once (and much worse also other processes so that we had to
>> reboot a server, essentially).
> I found that setting RLIMIT_AS [1] works very well on Linux. But this
> requires that you cap memory to some fixed value.
>
>> library(RAppArmor)
>> rlimit_as(1e9)
>> rnorm(1e9)
> Error: cannot allocate vector of size 7.5 Gb
>
> The RAppArmor package has many other utilities to protect your server
> such from a mis-behaving process such as limiting cpu time
> (RLIMIT_CPU), fork bombs (RLIMIT_NPROC) and file sizes (RLIMIT_FSIZE).
>
> [1] http://linux.die.net/man/2/getrlimit
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

>>>>> Kirill M?ller <kirill.mueller at ivt.baug.ethz.ch>
>>>>>     on Wed, 11 May 2016 10:42:56 +0200 writes:
    > My ulimit package exposes this API ([1], should finally submit it to 
    > CRAN); unfortunately this very API seems to be unsupported on OS X 
    > [2,3]. Last time I looked into it, neither of the documented settings 
    > achieved the desired effect.

    > -Kirill

    > [1] http://krlmlr.github.io/ulimit
    > [2] 
    >
http://stackoverflow.com/questions/3274385/how-to-limit-memory-of-a-os-x-program-ulimit-v-neither-m-are-working
    > [3] 
    >
https://developer.apple.com/library/ios/documentation/System/Conceptual/ManPages_iPhoneOS/man2/getrlimit.2.html


    > On 10.05.2016 01:08, Jeroen Ooms wrote:
    >> On 05/05/2016 10:11, Uwe Ligges wrote:
    >>> Actually this also happens under Linux and I had my R processes
killed
    >>> more than once (and much worse also other processes so that we
had to
    >>> reboot a server, essentially).

I agree that Linux is not consistently fine here either.

    >> I found that setting RLIMIT_AS [1] works very well on Linux. But
this
    >> requires that you cap memory to some fixed value.

conceivably one could set a default cap, using something
equivalent to the data in

		sfsmisc::Sys.meminfo()
or		sfsmisc::Sys.memGB()		

(very simple, > 10 year old interfaces, based on the  Linux-only (?)
 '/proc/*' filesystem).


In an ideal word, some of us,
    from R core, Jeroen, Kyrill, <one person knowing much about Windows>,
    maintainer("microbenchmark>, ...
would sit together and devise an R function interface (based on
low level platform specific interfaces, specifically for at least
Linux/POSIX-compliant, Mac, and Windows) which would allow
something  like your rlimit(..) calls below.

We'd really need something to work on all platforms ideally,
to be used by R package maintainers
and possibly even better by R itself at startup, setting a
reasonable memory cap - which the user could raise even to +Inf (or lower
even more).

Martin

---------
Final notes about RAppArmor, not relevant to the main thread topic :

Note: I'm working in pretty well maintained Fedora Linux env,
but Apparmor is not only not activated, but even not available.
OTOH, usting  RLIMIT / gerlimit on Linux is very generally
available.

As a consequence, the three last lines of

   > require(RAppArmor)
   Loading required package: RAppArmor
   Loading required package: tools
   Failed to lookup process confinement:
   AppArmor not supported on this system
   Have a look at: sudo aa-status

were very confusing to me:  My conclusion was I could not use
the RAppArmor package.

(But that's wrong: For the  rlimit*()  functions below, one do
 *NOT* need an AppArmor-enabled version of Linux !)
		  
    >>> library(RAppArmor)
    >>> rlimit_as(1e9)
    >>> rnorm(1e9)
    >> Error: cannot allocate vector of size 7.5 Gb
    >> 
    >> The RAppArmor package has many other utilities to protect your
server
    >> such from a mis-behaving process such as limiting cpu time
    >> (RLIMIT_CPU), fork bombs (RLIMIT_NPROC) and file sizes
(RLIMIT_FSIZE).
    >> 
    >> [1] http://linux.die.net/man/2/getrlimit

and from my current explorations I gather that all of these are
*not* Apparmor related... so could/should maybe rather migrate
into a lightweight package not mentioning AppArmor ?