R devel - Nov 2016 - creating a long list puts R in a state where many things stop working

Hi,

After I create a long list e.g. with

   x <- vector(mode="list", length=3e9)

many bad things start to happen e.g. some things stop working with a
spurious error message:

   gc()
   # Error in gc() :
   #   long vectors not supported yet: 
/home/hpages/src/R-3.3.2/src/main/memory.c:1137

   x_lens <- lengths(x)
   # Error in lengths(x) :
   #  long vectors not supported yet: 
/home/hpages/src/R-3.3.2/src/main/memory.c:1668

But then some of them work again:

   gc()
   #          used (Mb) gc trigger    (Mb)   max used    (Mb)
   # Ncells  57046  3.1     368000    19.7     350000    18.7
   # Vcells 138060  1.1 4320596678 32963.6 4500397915 34335.4

while others still fail but now with a different error message:

   x_lens <- lengths(x)
   # Error: evaluation nested too deeply: infinite recursion / 
options(expressions=)?

etc...

The more I go, the more weird things I see so clearly my session
got corrupted. Finally, and not too surprisingly, after playing a
little bit more, my session eventually crashed.

Thanks,
H.

 > sessionInfo()
R version 3.3.2 (2016-10-31)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 14.04.3 LTS

locale:
  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
  [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base


-- 
Herv? Pag?s

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpages at fredhutch.org
Phone:  (206) 667-5791
Fax:    (206) 667-1319

For reference, running your code in a build of R-devel with sanitizers:
> x <- vector(mode="list", length=3e9)
memory.c:2747:27: runtime error: signed integer overflow: 2147483647 + 1
cannot be represented in type 'R_len_t' (aka 'int')
SUMMARY: AddressSanitizer: undefined-behavior memory.c:2747:27 in
ASAN:DEADLYSIGNAL
==================================================================50159==ERROR:
AddressSanitizer: SEGV on unknown address 0x000000000000
(pc 0x00010b3452a7 bp 0x7fff5669d2b0 sp 0x7fff5669cd80 T0)
==50159==The signal is caused by a READ memory access.
==50159==Hint: address points to the zero page.
    #0 0x10b3452a6 in Rf_allocVector3 memory.c:2748
    #1 0x10afc13b1 in do_makevector builtin.c:789
    #2 0x10b238868 in bcEval eval.c:6041
    #3 0x10b2241b8 in Rf_eval eval.c:626
    #4 0x10b2794d4 in Rf_applyClosure eval.c:1160
    #5 0x10b224d16 in Rf_eval eval.c:742
    #6 0x10b286591 in do_set eval.c:2227
    #7 0x10b2246ae in Rf_eval eval.c:695
    #8 0x10b3229ef in Rf_ReplIteration main.c:258
    #9 0x10b327280 in R_ReplConsole main.c:308
    #10 0x10b327087 in run_Rmainloop main.c:1059
    #11 0x10955fea4 in main Rmain.c:29
    #12 0x7fffcabf3254 in start (libdyld.dylib+0x5254)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV memory.c:2748 in Rf_allocVector3
==50159==ABORTING
Abort trap: 6

Looks like there's an index here that needs to be R_xlen_t?
https://github.com/wch/r-source/blob/6e7a2ed989027f3800d2e2d64e60e6d700034c6b/src/main/memory.c#L2747

Declared at
https://github.com/wch/r-source/blob/6e7a2ed989027f3800d2e2d64e60e6d700034c6b/src/main/memory.c#L2476
-- could this index safely become R_xlen_t?

Kevin

On Mon, Nov 14, 2016 at 10:10 PM, Herv? Pag?s <hpages at fredhutch.org>
wrote:
> Hi,
>
> After I create a long list e.g. with
>
>   x <- vector(mode="list", length=3e9)
>
> many bad things start to happen e.g. some things stop working with a
> spurious error message:
>
>   gc()
>   # Error in gc() :
>   #   long vectors not supported yet: /home/hpages/src/R-3.3.2/src/m
> ain/memory.c:1137
>
>   x_lens <- lengths(x)
>   # Error in lengths(x) :
>   #  long vectors not supported yet: /home/hpages/src/R-3.3.2/src/m
> ain/memory.c:1668
>
> But then some of them work again:
>
>   gc()
>   #          used (Mb) gc trigger    (Mb)   max used    (Mb)
>   # Ncells  57046  3.1     368000    19.7     350000    18.7
>   # Vcells 138060  1.1 4320596678 32963.6 4500397915 34335.4
>
> while others still fail but now with a different error message:
>
>   x_lens <- lengths(x)
>   # Error: evaluation nested too deeply: infinite recursion /
> options(expressions=)?
>
> etc...
>
> The more I go, the more weird things I see so clearly my session
> got corrupted. Finally, and not too surprisingly, after playing a
> little bit more, my session eventually crashed.
>
> Thanks,
> H.
>
> > sessionInfo()
> R version 3.3.2 (2016-10-31)
> Platform: x86_64-pc-linux-gnu (64-bit)
> Running under: Ubuntu 14.04.3 LTS
>
> locale:
>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
>  [9] LC_ADDRESS=C               LC_TELEPHONE=C
> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
>
> --
> Herv? Pag?s
>
> Program in Computational Biology
> Division of Public Health Sciences
> Fred Hutchinson Cancer Research Center
> 1100 Fairview Ave. N, M1-B514
> P.O. Box 19024
> Seattle, WA 98109-1024
>
> E-mail: hpages at fredhutch.org
> Phone:  (206) 667-5791
> Fax:    (206) 667-1319
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
	[[alternative HTML version deleted]]