R devel - Apr 2018 - R Bug: write.table for matrix of more than 2, 147, 483, 648 elements

Hi Colton,

You could divide your write task into chunks that do not violate the 2^31-1
limit.

write.table has an append argument (default FALSE). 

Figure out a row chunk size nri < nr such that nri * nc is under 2^31-1 and
use
write.table() to write that out.

Then use 
write.table(  append = TRUE, )
for the next chunk of rows, looping over chunks until done.  Two chunks will get
your 2.8 billion entries done.


Best

Steve



Steven McKinney, Ph.D.

Statistician
Molecular Oncology and Breast Cancer Program
British Columbia Cancer Research Centre



> -----Original Message-----
> From: R-devel [mailto:r-devel-bounces at r-project.org] On Behalf Of
Tousey,
> Colton
> Sent: April-18-18 2:08 PM
> To: R-core at r-project.org; simon.urbanek at R-project.org; R-devel at r-
> project.org
> Subject: [Rd] R Bug: write.table for matrix of more than 2, 147, 483, 648
> elements
> 
> Hello,
> 
> I want to report a bug in R that is limiting my capabilities to export a
> matrix with write.csv or write.table with over 2,147,483,648 elements
(C's
> int limit). I found this bug already reported about before: https://bugs.r-
> project.org/bugzilla/show_bug.cgi?id=17182. However, there appears to be no
> solution or fixes in upcoming R version releases.
> 
> The error message is coming from the writetable part of the utils package
> in the io.c source code(https://svn.r-
> project.org/R/trunk/src/library/utils/src/io.c):
> /* quick integrity check */
>                 if(XLENGTH(x) != (R_len_t)nr * nc)
>                     error(_("corrupt matrix -- dims not not match
> length"));
> 
> The issue is that nr*nc is an integer and the size of my matrix, 2.8
> billion elements, exceeds C's limit, so the check forces the code to
fail.
> 
> My version:
> > R.Version()
> $platform
> [1] "x86_64-w64-mingw32"
> 
> $arch
> [1] "x86_64"
> 
> $os
> [1] "mingw32"
> 
> $system
> [1] "x86_64, mingw32"
> 
> $status
> [1] ""
> 
> $major
> [1] "3"
> 
> $minor
> [1] "4.3"
> 
> $year
> [1] "2017"
> 
> $month
> [1] "11"
> 
> $day
> [1] "30"
> 
> $`svn rev`
> [1] "73796"
> 
> $language
> [1] "R"
> 
> $version.string
> [1] "R version 3.4.3 (2017-11-30)"
> 
> $nickname
> [1] "Kite-Eating Tree"
> 
> Thank you,
> Colton
> 
> 
> Colton Tousey
> Research Associate II
> P: 816.585.0300   E: colton.tousey at kc.frb.org
> FEDERAL RESERVE BANK OF KANSAS CITY
> 1 Memorial Drive   *   Kansas City, Missouri 64198   *
> www.kansascityfed.org
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

On 2018-04-18 17:38, Steven McKinney wrote:
> Hi Colton,
>
> You could divide your write task into chunks that do not violate the 2^31-1
limit.
>
> write.table has an append argument (default FALSE).
>
> Figure out a row chunk size nri < nr such that nri * nc is under 2^31-1
and use
> write.table() to write that out.
>
> Then use
> write.table(  append = TRUE, )
> for the next chunk of rows, looping over chunks until done.  Two chunks
will get your 2.8 billion entries done.

 ????? Magnificent:? Is that something that could be implemented inside 
write.table?


 ????? Spencer
>
> Best
>
> Steve
>
>
>
> Steven McKinney, Ph.D.
>
> Statistician
> Molecular Oncology and Breast Cancer Program
> British Columbia Cancer Research Centre
>
>
>
>
>> -----Original Message-----
>> From: R-devel [mailto:r-devel-bounces at r-project.org] On Behalf Of
Tousey,
>> Colton
>> Sent: April-18-18 2:08 PM
>> To: R-core at r-project.org; simon.urbanek at R-project.org; R-devel at
r-
>> project.org
>> Subject: [Rd] R Bug: write.table for matrix of more than 2, 147, 483,
648
>> elements
>>
>> Hello,
>>
>> I want to report a bug in R that is limiting my capabilities to export
a
>> matrix with write.csv or write.table with over 2,147,483,648 elements
(C's
>> int limit). I found this bug already reported about before:
https://bugs.r-
>> project.org/bugzilla/show_bug.cgi?id=17182. However, there appears to
be no
>> solution or fixes in upcoming R version releases.
>>
>> The error message is coming from the writetable part of the utils
package
>> in the io.c source code(https://svn.r-
>> project.org/R/trunk/src/library/utils/src/io.c):
>> /* quick integrity check */
>>                  if(XLENGTH(x) != (R_len_t)nr * nc)
>>                      error(_("corrupt matrix -- dims not not match
>> length"));
>>
>> The issue is that nr*nc is an integer and the size of my matrix, 2.8
>> billion elements, exceeds C's limit, so the check forces the code
to fail.
>>
>> My version:
>>> R.Version()
>> $platform
>> [1] "x86_64-w64-mingw32"
>>
>> $arch
>> [1] "x86_64"
>>
>> $os
>> [1] "mingw32"
>>
>> $system
>> [1] "x86_64, mingw32"
>>
>> $status
>> [1] ""
>>
>> $major
>> [1] "3"
>>
>> $minor
>> [1] "4.3"
>>
>> $year
>> [1] "2017"
>>
>> $month
>> [1] "11"
>>
>> $day
>> [1] "30"
>>
>> $`svn rev`
>> [1] "73796"
>>
>> $language
>> [1] "R"
>>
>> $version.string
>> [1] "R version 3.4.3 (2017-11-30)"
>>
>> $nickname
>> [1] "Kite-Eating Tree"
>>
>> Thank you,
>> Colton
>>
>>
>> Colton Tousey
>> Research Associate II
>> P: 816.585.0300   E: colton.tousey at kc.frb.org
>> FEDERAL RESERVE BANK OF KANSAS CITY
>> 1 Memorial Drive   *   Kansas City, Missouri 64198   *
>> www.kansascityfed.org
>>
>> 	[[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel