R devel - May 2016 - R process killed when allocating too large matrix (Mac OS X)

>>>>> Kirill M?ller <kirill.mueller at ivt.baug.ethz.ch>
>>>>>     on Wed, 11 May 2016 10:42:56 +0200 writes:
    > My ulimit package exposes this API ([1], should finally submit it to 
    > CRAN); unfortunately this very API seems to be unsupported on OS X 
    > [2,3]. Last time I looked into it, neither of the documented settings 
    > achieved the desired effect.

    > -Kirill

    > [1] http://krlmlr.github.io/ulimit
    > [2] 
    >
http://stackoverflow.com/questions/3274385/how-to-limit-memory-of-a-os-x-program-ulimit-v-neither-m-are-working
    > [3] 
    >
https://developer.apple.com/library/ios/documentation/System/Conceptual/ManPages_iPhoneOS/man2/getrlimit.2.html


    > On 10.05.2016 01:08, Jeroen Ooms wrote:
    >> On 05/05/2016 10:11, Uwe Ligges wrote:
    >>> Actually this also happens under Linux and I had my R processes
killed
    >>> more than once (and much worse also other processes so that we
had to
    >>> reboot a server, essentially).

I agree that Linux is not consistently fine here either.

    >> I found that setting RLIMIT_AS [1] works very well on Linux. But
this
    >> requires that you cap memory to some fixed value.

conceivably one could set a default cap, using something
equivalent to the data in

		sfsmisc::Sys.meminfo()
or		sfsmisc::Sys.memGB()		

(very simple, > 10 year old interfaces, based on the  Linux-only (?)
 '/proc/*' filesystem).


In an ideal word, some of us,
    from R core, Jeroen, Kyrill, <one person knowing much about Windows>,
    maintainer("microbenchmark>, ...
would sit together and devise an R function interface (based on
low level platform specific interfaces, specifically for at least
Linux/POSIX-compliant, Mac, and Windows) which would allow
something  like your rlimit(..) calls below.

We'd really need something to work on all platforms ideally,
to be used by R package maintainers
and possibly even better by R itself at startup, setting a
reasonable memory cap - which the user could raise even to +Inf (or lower
even more).

Martin

---------
Final notes about RAppArmor, not relevant to the main thread topic :

Note: I'm working in pretty well maintained Fedora Linux env,
but Apparmor is not only not activated, but even not available.
OTOH, usting  RLIMIT / gerlimit on Linux is very generally
available.

As a consequence, the three last lines of

   > require(RAppArmor)
   Loading required package: RAppArmor
   Loading required package: tools
   Failed to lookup process confinement:
   AppArmor not supported on this system
   Have a look at: sudo aa-status

were very confusing to me:  My conclusion was I could not use
the RAppArmor package.

(But that's wrong: For the  rlimit*()  functions below, one do
 *NOT* need an AppArmor-enabled version of Linux !)
		  
    >>> library(RAppArmor)
    >>> rlimit_as(1e9)
    >>> rnorm(1e9)
    >> Error: cannot allocate vector of size 7.5 Gb
    >> 
    >> The RAppArmor package has many other utilities to protect your
server
    >> such from a mis-behaving process such as limiting cpu time
    >> (RLIMIT_CPU), fork bombs (RLIMIT_NPROC) and file sizes
(RLIMIT_FSIZE).
    >> 
    >> [1] http://linux.die.net/man/2/getrlimit

and from my current explorations I gather that all of these are
*not* Apparmor related... so could/should maybe rather migrate
into a lightweight package not mentioning AppArmor ?

On Thu, May 12, 2016 at 9:51 AM, Martin Maechler
<maechler at stat.math.ethz.ch> wrote:
> My conclusion was I could not use the RAppArmor package.
>
> (But that's wrong: For the  rlimit*()  functions below, one do
>  *NOT* need an AppArmor-enabled version of Linux !)
Yes, it is a relatively recent (unadvertised) feature that the package
now builds on linux systems without libapparmor. I agree this names
the package name confusing. I'll make at least that warning more
informative.

Some background: When I started the package (5 years ago) I expected
that soon all linux distributions would have the apparmor module which
has been in the kernel since 2.6.36. However Redhat is explicitly
disabling it because they are pushing a competing MAC system (SELinux)
which they develop together with the NSA, and they really want you to
use this instead (and only this!).
> I gather that all of these are *not* Apparmor related... so could/should
maybe rather migrate into a lightweight package not mentioning AppArmor ?
I agree, it has been on the to do list for a while; Kirill and me were
talking yesterday about what would be the best route to take:

 - A small package with only the rlimit bindings
 - or: A 'linux' package with bindings to anything in the kernel,
including rlimit, but possibly other system tools.
 - or: A package targeting POSIX/unix with standard functionality that
is also available on OSX/BSD.
>From my experience, windows is pretty useless for this kind of stuff.

On Thu, 12 May 2016 12:07:29 +0200
Jeroen Ooms <jeroen.ooms at stat.ucla.edu> wrote:
> On Thu, May 12, 2016 at 9:51 AM, Martin Maechler
> <maechler at stat.math.ethz.ch> wrote:
> > My conclusion was I could not use the RAppArmor package.
> >
> > (But that's wrong: For the  rlimit*()  functions below, one do
> >  *NOT* need an AppArmor-enabled version of Linux !)  
> 
> Yes, it is a relatively recent (unadvertised) feature that the package
> now builds on linux systems without libapparmor. I agree this names
> the package name confusing. I'll make at least that warning more
> informative.
> 
> Some background: When I started the package (5 years ago) I expected
> that soon all linux distributions would have the apparmor module which
> has been in the kernel since 2.6.36. However Redhat is explicitly
> disabling it because they are pushing a competing MAC system (SELinux)
> which they develop together with the NSA, and they really want you to
> use this instead (and only this!).
> 
> > I gather that all of these are *not* Apparmor related... so
> > could/should maybe rather migrate into a lightweight package not
> > mentioning AppArmor ?  
> 
> I agree, it has been on the to do list for a while; Kirill and me were
> talking yesterday about what would be the best route to take:
> 
>  - A small package with only the rlimit bindings
>  - or: A 'linux' package with bindings to anything in the kernel,
> including rlimit, but possibly other system tools.
>  - or: A package targeting POSIX/unix with standard functionality that
> is also available on OSX/BSD.
> 
> >From my experience, windows is pretty useless for this kind of
> >stuff.  
Maybe not so useless after reading [1] about computationally
querying the OSes on the available memory and [2] about pushing the
OSes' limits. The latter page is part of a series where each topic is
valuable on its own.

[1]?https://msdn.microsoft.com/en-us/library/aa366778.aspx
[2]?https://blogs.technet.microsoft.com/markrussinovich/2008/07/21/pushing-the-limits-of-windows-physical-memory/
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

On 12.05.2016 09:51, Martin Maechler wrote:
>      > My ulimit package exposes this API ([1], should finally submit it
to
>      > CRAN); unfortunately this very API seems to be unsupported on OS
X
>      > [2,3]. Last time I looked into it, neither of the documented
settings
>      > achieved the desired effect.
>
>      > -Kirill
>
>      > [1] http://krlmlr.github.io/ulimit
>      > [2]
>      >
http://stackoverflow.com/questions/3274385/how-to-limit-memory-of-a-os-x-program-ulimit-v-neither-m-are-working
>      > [3]
>      >
https://developer.apple.com/library/ios/documentation/System/Conceptual/ManPages_iPhoneOS/man2/getrlimit.2.html
>
>
> ...
>
> In an ideal word, some of us,
>      from R core, Jeroen, Kyrill, <one person knowing much about
Windows>,
>      maintainer("microbenchmark>, ...
> would sit together and devise an R function interface (based on
> low level platform specific interfaces, specifically for at least
> Linux/POSIX-compliant, Mac, and Windows) which would allow
> something  like your rlimit(..) calls below.
>
> We'd really need something to work on all platforms ideally,
> to be used by R package maintainers
> and possibly even better by R itself at startup, setting a
> reasonable memory cap - which the user could raise even to +Inf (or lower
> even more).
>
I haven't found a Windows API that allows limiting the address space, 
only one that limits the working set size; it seems likely that this is 
the best we can get on OS X, too, but then my experience with OS X is 
very limited.

mallinfo() is used on Windows and seems to be available on Linux, too, 
but not on OS X.


-Kirill