R devel - May 2016 - R process killed when allocating too large matrix (Mac OS X)

Hi Simon,

thanks for your quick reply.

1) ... so you can reproduce this?
2) Do you know a way how this can be 'foreseen'? We allocate larger
matrices in the copula package depending on the user's input
dimension. It would be good to tell her/him "Your dimension is quite
large. Be aware of killers in your neighborhood"... before the killer
attacks.

Thanks & cheers,
Marius


On Wed, May 4, 2016 at 8:54 PM, Simon Urbanek
<simon.urbanek at r-project.org> wrote:
>
> On May 4, 2016, at 6:14 PM, Marius Hofert <marius.hofert at
uwaterloo.ca> wrote:
>
>>> Can you elaborate on "leads to R being killed"? You
should tell to the killer not to do it again :).
>>
>> Hi Simon!
>>
>> Sure, but who do you tell it if you don't know the killer?
>> This is all the killer left me with, the 'crime scene' if you
like :-)
>>
>>> m <- matrix(0, 90000, 100000)
>> Killed: 9
>>
>> My colleague Wayne Oldford also tried it on his Mac machine and
>> apparently the killer went further down the hallway to his office
>> now... so scary. Here is Wayne's sessionInfo():
>>
>
> Yes, indeed, scary - since it means someone is killing R which means there
is not much R itself can do about it. In fact from the syslog I see
>
> May  4 20:48:11 ginaz kernel[0]: low swap: killing pid 56256 (R)
>
> so it's the kernel's own defense mechanism. The bad thing is that R
cannot do anything about it - the kernel just decides to snipe processes it
thinks are dangerous to the health of the system, and it does so without a
warning.
>
> Cheers,
> Simon
>

On May 4, 2016, at 9:00 PM, Marius Hofert <marius.hofert at uwaterloo.ca>
wrote:
> Hi Simon,
> 
> thanks for your quick reply.
> 
> 1) ... so you can reproduce this?
Yes, I can on 10.11.4.

> 2) Do you know a way how this can be 'foreseen'? We allocate larger
> matrices in the copula package depending on the user's input
> dimension. It would be good to tell her/him "Your dimension is quite
> large. Be aware of killers in your neighborhood"... before the killer
> attacks.
> 
Not directly, because the system is happy to accommodate R's request for
more memory but then it will strike out of the blue. Since that decision is made
by the kernel and it doesn't expose its thinking about how it feels,
it's hard to tell. I couldn't find any API on OS X akin to
didReceiveMemoryWarning in iOS, unfortunately.

You could guess - e.g. by checking the total system memory

int mib [] = { CTL_HW, HW_MEMSIZE };
int64_t value = 0;
size_t length = sizeof(value);
sysctl(mib, 2, &value, &length, NULL, 0);

and comparing it to the sizes involved. However, even that is not foolproof,
because it all depends on the other processes' memory usage, swap space size
etc. There are slightly more involved ways to query the VM system as well, but
I'm not sure I'd want to go so deep into the weeds, especially since
this becomes quickly highly OS-specific.

Cheers,
Simon


> Thanks & cheers,
> Marius
> 
> 
> On Wed, May 4, 2016 at 8:54 PM, Simon Urbanek
> <simon.urbanek at r-project.org> wrote:
>> 
>> On May 4, 2016, at 6:14 PM, Marius Hofert <marius.hofert at
uwaterloo.ca> wrote:
>> 
>>>> Can you elaborate on "leads to R being killed"? You
should tell to the killer not to do it again :).
>>> 
>>> Hi Simon!
>>> 
>>> Sure, but who do you tell it if you don't know the killer?
>>> This is all the killer left me with, the 'crime scene' if
you like :-)
>>> 
>>>> m <- matrix(0, 90000, 100000)
>>> Killed: 9
>>> 
>>> My colleague Wayne Oldford also tried it on his Mac machine and
>>> apparently the killer went further down the hallway to his office
>>> now... so scary. Here is Wayne's sessionInfo():
>>> 
>> 
>> Yes, indeed, scary - since it means someone is killing R which means
there is not much R itself can do about it. In fact from the syslog I see
>> 
>> May  4 20:48:11 ginaz kernel[0]: low swap: killing pid 56256 (R)
>> 
>> so it's the kernel's own defense mechanism. The bad thing is
that R cannot do anything about it - the kernel just decides to snipe processes
it thinks are dangerous to the health of the system, and it does so without a
warning.
>> 
>> Cheers,
>> Simon
>> 
>

Hi,

Interesting "feature" in 10.11.4. I wonder if the process is killed
before or after malloc() returns. If before, it seems very blunt:
"You're asking too much and I don't like it so I kill you
now".
If after it doesn't look much better: "You're asking a lot and I
don't like it but I give it to you anyway. I'll kill you quickly
later".

Why wouldn't the kernel just refuse to give that memory instead
(i.e. malloc() returns NULL).

Just curious...

H.

On 05/04/2016 06:21 PM, Simon Urbanek wrote:
>
> On May 4, 2016, at 9:00 PM, Marius Hofert <marius.hofert at
uwaterloo.ca> wrote:
>
>> Hi Simon,
>>
>> thanks for your quick reply.
>>
>> 1) ... so you can reproduce this?
>
> Yes, I can on 10.11.4.
>
>
>> 2) Do you know a way how this can be 'foreseen'? We allocate
larger
>> matrices in the copula package depending on the user's input
>> dimension. It would be good to tell her/him "Your dimension is
quite
>> large. Be aware of killers in your neighborhood"... before the
killer
>> attacks.
>>
>
> Not directly, because the system is happy to accommodate R's request
for more memory but then it will strike out of the blue. Since that decision is
made by the kernel and it doesn't expose its thinking about how it feels,
it's hard to tell. I couldn't find any API on OS X akin to
didReceiveMemoryWarning in iOS, unfortunately.
>
> You could guess - e.g. by checking the total system memory
>
> int mib [] = { CTL_HW, HW_MEMSIZE };
> int64_t value = 0;
> size_t length = sizeof(value);
> sysctl(mib, 2, &value, &length, NULL, 0);
>
> and comparing it to the sizes involved. However, even that is not
foolproof, because it all depends on the other processes' memory usage, swap
space size etc. There are slightly more involved ways to query the VM system as
well, but I'm not sure I'd want to go so deep into the weeds, especially
since this becomes quickly highly OS-specific.
>
> Cheers,
> Simon
>
>
>
>> Thanks & cheers,
>> Marius
>>
>>
>> On Wed, May 4, 2016 at 8:54 PM, Simon Urbanek
>> <simon.urbanek at r-project.org> wrote:
>>>
>>> On May 4, 2016, at 6:14 PM, Marius Hofert <marius.hofert at
uwaterloo.ca> wrote:
>>>
>>>>> Can you elaborate on "leads to R being killed"?
You should tell to the killer not to do it again :).
>>>>
>>>> Hi Simon!
>>>>
>>>> Sure, but who do you tell it if you don't know the killer?
>>>> This is all the killer left me with, the 'crime scene'
if you like :-)
>>>>
>>>>> m <- matrix(0, 90000, 100000)
>>>> Killed: 9
>>>>
>>>> My colleague Wayne Oldford also tried it on his Mac machine and
>>>> apparently the killer went further down the hallway to his
office
>>>> now... so scary. Here is Wayne's sessionInfo():
>>>>
>>>
>>> Yes, indeed, scary - since it means someone is killing R which
means there is not much R itself can do about it. In fact from the syslog I see
>>>
>>> May  4 20:48:11 ginaz kernel[0]: low swap: killing pid 56256 (R)
>>>
>>> so it's the kernel's own defense mechanism. The bad thing
is that R cannot do anything about it - the kernel just decides to snipe
processes it thinks are dangerous to the health of the system, and it does so
without a warning.
>>>
>>> Cheers,
>>> Simon
>>>
>>
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
-- 
Herv? Pag?s

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpages at fredhutch.org
Phone:  (206) 667-5791
Fax:    (206) 667-1319

Hi Simon,

... all interesting (but quite a bit above my head). I only read
'Linux' and want to throw in that this problem does not appear on
Linux (it seems). I talked about this with Martin Maechler and he
reported that the same example (on one of his machines; with NA_real_
instead of '0's in the matrix) gave:

  Error: cannot allocate vector of size 70.8 Gb
    Timing stopped at: 144.79 41.619 202.019

... but no killer around...

Cheers,
Marius

On 05.05.2016 04:25, Marius Hofert wrote:
> Hi Simon,
>
> ... all interesting (but quite a bit above my head). I only read
> 'Linux' and want to throw in that this problem does not appear on
> Linux (it seems). I talked about this with Martin Maechler and he
> reported that the same example (on one of his machines; with NA_real_
> instead of '0's in the matrix) gave:
>
>   Error: cannot allocate vector of size 70.8 Gb
>     Timing stopped at: 144.79 41.619 202.019
>
> ... but no killer around...
Well, with n=1. ;-)

Actually this also happens under Linux and I had my R processes killed 
more than once (and much worse also other processes so that we had to 
reboot a server, essentially). That's why we use job scheduling on 
servers for R nowadays ...

Best,
Uwe
>
> Cheers,
> Marius
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>