Marius Hofert
2016-May-05 01:00 UTC
[Rd] R process killed when allocating too large matrix (Mac OS X)
Hi Simon, thanks for your quick reply. 1) ... so you can reproduce this? 2) Do you know a way how this can be 'foreseen'? We allocate larger matrices in the copula package depending on the user's input dimension. It would be good to tell her/him "Your dimension is quite large. Be aware of killers in your neighborhood"... before the killer attacks. Thanks & cheers, Marius On Wed, May 4, 2016 at 8:54 PM, Simon Urbanek <simon.urbanek at r-project.org> wrote:> > On May 4, 2016, at 6:14 PM, Marius Hofert <marius.hofert at uwaterloo.ca> wrote: > >>> Can you elaborate on "leads to R being killed"? You should tell to the killer not to do it again :). >> >> Hi Simon! >> >> Sure, but who do you tell it if you don't know the killer? >> This is all the killer left me with, the 'crime scene' if you like :-) >> >>> m <- matrix(0, 90000, 100000) >> Killed: 9 >> >> My colleague Wayne Oldford also tried it on his Mac machine and >> apparently the killer went further down the hallway to his office >> now... so scary. Here is Wayne's sessionInfo(): >> > > Yes, indeed, scary - since it means someone is killing R which means there is not much R itself can do about it. In fact from the syslog I see > > May 4 20:48:11 ginaz kernel[0]: low swap: killing pid 56256 (R) > > so it's the kernel's own defense mechanism. The bad thing is that R cannot do anything about it - the kernel just decides to snipe processes it thinks are dangerous to the health of the system, and it does so without a warning. > > Cheers, > Simon >
Simon Urbanek
2016-May-05 01:21 UTC
[Rd] R process killed when allocating too large matrix (Mac OS X)
On May 4, 2016, at 9:00 PM, Marius Hofert <marius.hofert at uwaterloo.ca> wrote:> Hi Simon, > > thanks for your quick reply. > > 1) ... so you can reproduce this?Yes, I can on 10.11.4.> 2) Do you know a way how this can be 'foreseen'? We allocate larger > matrices in the copula package depending on the user's input > dimension. It would be good to tell her/him "Your dimension is quite > large. Be aware of killers in your neighborhood"... before the killer > attacks. >Not directly, because the system is happy to accommodate R's request for more memory but then it will strike out of the blue. Since that decision is made by the kernel and it doesn't expose its thinking about how it feels, it's hard to tell. I couldn't find any API on OS X akin to didReceiveMemoryWarning in iOS, unfortunately. You could guess - e.g. by checking the total system memory int mib [] = { CTL_HW, HW_MEMSIZE }; int64_t value = 0; size_t length = sizeof(value); sysctl(mib, 2, &value, &length, NULL, 0); and comparing it to the sizes involved. However, even that is not foolproof, because it all depends on the other processes' memory usage, swap space size etc. There are slightly more involved ways to query the VM system as well, but I'm not sure I'd want to go so deep into the weeds, especially since this becomes quickly highly OS-specific. Cheers, Simon> Thanks & cheers, > Marius > > > On Wed, May 4, 2016 at 8:54 PM, Simon Urbanek > <simon.urbanek at r-project.org> wrote: >> >> On May 4, 2016, at 6:14 PM, Marius Hofert <marius.hofert at uwaterloo.ca> wrote: >> >>>> Can you elaborate on "leads to R being killed"? You should tell to the killer not to do it again :). >>> >>> Hi Simon! >>> >>> Sure, but who do you tell it if you don't know the killer? >>> This is all the killer left me with, the 'crime scene' if you like :-) >>> >>>> m <- matrix(0, 90000, 100000) >>> Killed: 9 >>> >>> My colleague Wayne Oldford also tried it on his Mac machine and >>> apparently the killer went further down the hallway to his office >>> now... so scary. Here is Wayne's sessionInfo(): >>> >> >> Yes, indeed, scary - since it means someone is killing R which means there is not much R itself can do about it. In fact from the syslog I see >> >> May 4 20:48:11 ginaz kernel[0]: low swap: killing pid 56256 (R) >> >> so it's the kernel's own defense mechanism. The bad thing is that R cannot do anything about it - the kernel just decides to snipe processes it thinks are dangerous to the health of the system, and it does so without a warning. >> >> Cheers, >> Simon >> >
Hervé Pagès
2016-May-05 01:49 UTC
[Rd] R process killed when allocating too large matrix (Mac OS X)
Hi, Interesting "feature" in 10.11.4. I wonder if the process is killed before or after malloc() returns. If before, it seems very blunt: "You're asking too much and I don't like it so I kill you now". If after it doesn't look much better: "You're asking a lot and I don't like it but I give it to you anyway. I'll kill you quickly later". Why wouldn't the kernel just refuse to give that memory instead (i.e. malloc() returns NULL). Just curious... H. On 05/04/2016 06:21 PM, Simon Urbanek wrote:> > On May 4, 2016, at 9:00 PM, Marius Hofert <marius.hofert at uwaterloo.ca> wrote: > >> Hi Simon, >> >> thanks for your quick reply. >> >> 1) ... so you can reproduce this? > > Yes, I can on 10.11.4. > > >> 2) Do you know a way how this can be 'foreseen'? We allocate larger >> matrices in the copula package depending on the user's input >> dimension. It would be good to tell her/him "Your dimension is quite >> large. Be aware of killers in your neighborhood"... before the killer >> attacks. >> > > Not directly, because the system is happy to accommodate R's request for more memory but then it will strike out of the blue. Since that decision is made by the kernel and it doesn't expose its thinking about how it feels, it's hard to tell. I couldn't find any API on OS X akin to didReceiveMemoryWarning in iOS, unfortunately. > > You could guess - e.g. by checking the total system memory > > int mib [] = { CTL_HW, HW_MEMSIZE }; > int64_t value = 0; > size_t length = sizeof(value); > sysctl(mib, 2, &value, &length, NULL, 0); > > and comparing it to the sizes involved. However, even that is not foolproof, because it all depends on the other processes' memory usage, swap space size etc. There are slightly more involved ways to query the VM system as well, but I'm not sure I'd want to go so deep into the weeds, especially since this becomes quickly highly OS-specific. > > Cheers, > Simon > > > >> Thanks & cheers, >> Marius >> >> >> On Wed, May 4, 2016 at 8:54 PM, Simon Urbanek >> <simon.urbanek at r-project.org> wrote: >>> >>> On May 4, 2016, at 6:14 PM, Marius Hofert <marius.hofert at uwaterloo.ca> wrote: >>> >>>>> Can you elaborate on "leads to R being killed"? You should tell to the killer not to do it again :). >>>> >>>> Hi Simon! >>>> >>>> Sure, but who do you tell it if you don't know the killer? >>>> This is all the killer left me with, the 'crime scene' if you like :-) >>>> >>>>> m <- matrix(0, 90000, 100000) >>>> Killed: 9 >>>> >>>> My colleague Wayne Oldford also tried it on his Mac machine and >>>> apparently the killer went further down the hallway to his office >>>> now... so scary. Here is Wayne's sessionInfo(): >>>> >>> >>> Yes, indeed, scary - since it means someone is killing R which means there is not much R itself can do about it. In fact from the syslog I see >>> >>> May 4 20:48:11 ginaz kernel[0]: low swap: killing pid 56256 (R) >>> >>> so it's the kernel's own defense mechanism. The bad thing is that R cannot do anything about it - the kernel just decides to snipe processes it thinks are dangerous to the health of the system, and it does so without a warning. >>> >>> Cheers, >>> Simon >>> >> > > ______________________________________________ > R-devel at r-project.org mailing list > stat.ethz.ch/mailman/listinfo/r-devel >-- Herv? Pag?s Program in Computational Biology Division of Public Health Sciences Fred Hutchinson Cancer Research Center 1100 Fairview Ave. N, M1-B514 P.O. Box 19024 Seattle, WA 98109-1024 E-mail: hpages at fredhutch.org Phone: (206) 667-5791 Fax: (206) 667-1319
Marius Hofert
2016-May-05 02:25 UTC
[Rd] R process killed when allocating too large matrix (Mac OS X)
Hi Simon, ... all interesting (but quite a bit above my head). I only read 'Linux' and want to throw in that this problem does not appear on Linux (it seems). I talked about this with Martin Maechler and he reported that the same example (on one of his machines; with NA_real_ instead of '0's in the matrix) gave: Error: cannot allocate vector of size 70.8 Gb Timing stopped at: 144.79 41.619 202.019 ... but no killer around... Cheers, Marius
Uwe Ligges
2016-May-05 09:11 UTC
[Rd] R process killed when allocating too large matrix (Mac OS X)
On 05.05.2016 04:25, Marius Hofert wrote:> Hi Simon, > > ... all interesting (but quite a bit above my head). I only read > 'Linux' and want to throw in that this problem does not appear on > Linux (it seems). I talked about this with Martin Maechler and he > reported that the same example (on one of his machines; with NA_real_ > instead of '0's in the matrix) gave: > > Error: cannot allocate vector of size 70.8 Gb > Timing stopped at: 144.79 41.619 202.019 > > ... but no killer around...Well, with n=1. ;-) Actually this also happens under Linux and I had my R processes killed more than once (and much worse also other processes so that we had to reboot a server, essentially). That's why we use job scheduling on servers for R nowadays ... Best, Uwe> > Cheers, > Marius > > ______________________________________________ > R-devel at r-project.org mailing list > stat.ethz.ch/mailman/listinfo/r-devel >
Apparently Analagous Threads
- R process killed when allocating too large matrix (Mac OS X)
- R process killed when allocating too large matrix (Mac OS X)
- R process killed when allocating too large matrix (Mac OS X)
- R process killed when allocating too large matrix (Mac OS X)
- R process killed when allocating too large matrix (Mac OS X)