Hervé Pagès
2016-Nov-15 06:10 UTC
[Rd] creating a long list puts R in a state where many things stop working
Hi, After I create a long list e.g. with x <- vector(mode="list", length=3e9) many bad things start to happen e.g. some things stop working with a spurious error message: gc() # Error in gc() : # long vectors not supported yet: /home/hpages/src/R-3.3.2/src/main/memory.c:1137 x_lens <- lengths(x) # Error in lengths(x) : # long vectors not supported yet: /home/hpages/src/R-3.3.2/src/main/memory.c:1668 But then some of them work again: gc() # used (Mb) gc trigger (Mb) max used (Mb) # Ncells 57046 3.1 368000 19.7 350000 18.7 # Vcells 138060 1.1 4320596678 32963.6 4500397915 34335.4 while others still fail but now with a different error message: x_lens <- lengths(x) # Error: evaluation nested too deeply: infinite recursion / options(expressions=)? etc... The more I go, the more weird things I see so clearly my session got corrupted. Finally, and not too surprisingly, after playing a little bit more, my session eventually crashed. Thanks, H. > sessionInfo() R version 3.3.2 (2016-10-31) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu 14.04.3 LTS locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 [7] LC_PAPER=en_US.UTF-8 LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] stats graphics grDevices utils datasets methods base -- Herv? Pag?s Program in Computational Biology Division of Public Health Sciences Fred Hutchinson Cancer Research Center 1100 Fairview Ave. N, M1-B514 P.O. Box 19024 Seattle, WA 98109-1024 E-mail: hpages at fredhutch.org Phone: (206) 667-5791 Fax: (206) 667-1319
Kevin Ushey
2016-Nov-15 20:26 UTC
[Rd] creating a long list puts R in a state where many things stop working
For reference, running your code in a build of R-devel with sanitizers:> x <- vector(mode="list", length=3e9)memory.c:2747:27: runtime error: signed integer overflow: 2147483647 + 1 cannot be represented in type 'R_len_t' (aka 'int') SUMMARY: AddressSanitizer: undefined-behavior memory.c:2747:27 in ASAN:DEADLYSIGNAL ==================================================================50159==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x00010b3452a7 bp 0x7fff5669d2b0 sp 0x7fff5669cd80 T0) ==50159==The signal is caused by a READ memory access. ==50159==Hint: address points to the zero page. #0 0x10b3452a6 in Rf_allocVector3 memory.c:2748 #1 0x10afc13b1 in do_makevector builtin.c:789 #2 0x10b238868 in bcEval eval.c:6041 #3 0x10b2241b8 in Rf_eval eval.c:626 #4 0x10b2794d4 in Rf_applyClosure eval.c:1160 #5 0x10b224d16 in Rf_eval eval.c:742 #6 0x10b286591 in do_set eval.c:2227 #7 0x10b2246ae in Rf_eval eval.c:695 #8 0x10b3229ef in Rf_ReplIteration main.c:258 #9 0x10b327280 in R_ReplConsole main.c:308 #10 0x10b327087 in run_Rmainloop main.c:1059 #11 0x10955fea4 in main Rmain.c:29 #12 0x7fffcabf3254 in start (libdyld.dylib+0x5254) AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: SEGV memory.c:2748 in Rf_allocVector3 ==50159==ABORTING Abort trap: 6 Looks like there's an index here that needs to be R_xlen_t? https://github.com/wch/r-source/blob/6e7a2ed989027f3800d2e2d64e60e6d700034c6b/src/main/memory.c#L2747 Declared at https://github.com/wch/r-source/blob/6e7a2ed989027f3800d2e2d64e60e6d700034c6b/src/main/memory.c#L2476 -- could this index safely become R_xlen_t? Kevin On Mon, Nov 14, 2016 at 10:10 PM, Herv? Pag?s <hpages at fredhutch.org> wrote:> Hi, > > After I create a long list e.g. with > > x <- vector(mode="list", length=3e9) > > many bad things start to happen e.g. some things stop working with a > spurious error message: > > gc() > # Error in gc() : > # long vectors not supported yet: /home/hpages/src/R-3.3.2/src/m > ain/memory.c:1137 > > x_lens <- lengths(x) > # Error in lengths(x) : > # long vectors not supported yet: /home/hpages/src/R-3.3.2/src/m > ain/memory.c:1668 > > But then some of them work again: > > gc() > # used (Mb) gc trigger (Mb) max used (Mb) > # Ncells 57046 3.1 368000 19.7 350000 18.7 > # Vcells 138060 1.1 4320596678 32963.6 4500397915 34335.4 > > while others still fail but now with a different error message: > > x_lens <- lengths(x) > # Error: evaluation nested too deeply: infinite recursion / > options(expressions=)? > > etc... > > The more I go, the more weird things I see so clearly my session > got corrupted. Finally, and not too surprisingly, after playing a > little bit more, my session eventually crashed. > > Thanks, > H. > > > sessionInfo() > R version 3.3.2 (2016-10-31) > Platform: x86_64-pc-linux-gnu (64-bit) > Running under: Ubuntu 14.04.3 LTS > > locale: > [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C > [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 > [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 > [7] LC_PAPER=en_US.UTF-8 LC_NAME=C > [9] LC_ADDRESS=C LC_TELEPHONE=C > [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > > -- > Herv? Pag?s > > Program in Computational Biology > Division of Public Health Sciences > Fred Hutchinson Cancer Research Center > 1100 Fairview Ave. N, M1-B514 > P.O. Box 19024 > Seattle, WA 98109-1024 > > E-mail: hpages at fredhutch.org > Phone: (206) 667-5791 > Fax: (206) 667-1319 > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel[[alternative HTML version deleted]]