Posting for a friend
Begin forwarded message:
From: "Geier, Florian"
<florian.geier08@imperial.ac.uk<mailto:florian.geier08@imperial.ac.uk>>
Subject: Fwd: readBin fails to read large files
Date: September 1, 2011 4:10:53 PM GMT+01:00
To:
Begin forwarded message:
Date: 1 September 2011 16:01:45 GMT+01:00
Subject: readBin fails to read large files
Dear all,
I am trying to read a large file (~2GB) of unsigned ints into R. Using the
command:
raw<-readBin("file",n=10^8,
integer(),endian="little",signed=FALSE)
It works fine for n=10^8, but fails for n=10^9 (or even at n=6*10^8). My
machine$sizeof.long is 8 bit.
I am running R 2.13.1 on a x86_64-apple-darwin9.8.0/x86_64 (64-bit)
architecture.
Thanks for your help
Florian
--
AXA doctoral fellow
Bundy lab - Biomolecular Medicine
Imperial College London
--
AXA doctoral fellow
Bundy lab - Biomolecular Medicine
Imperial College London
[[alternative HTML version deleted]]
Are you running a 64-bit version of R? It sounds like your operating system is not giving you enough memory. It looks like this is not under Windows in a native mode. On Thu, Sep 1, 2011 at 11:13 AM, Benton, Paul <hpaul.benton08 at imperial.ac.uk> wrote:> Posting for a friend > > Begin forwarded message: > > From: "Geier, Florian" <florian.geier08 at imperial.ac.uk<mailto:florian.geier08 at imperial.ac.uk>> > Subject: Fwd: readBin fails to read large files > Date: September 1, 2011 4:10:53 PM GMT+01:00 > To: > > > > Begin forwarded message: > > Date: 1 September 2011 16:01:45 GMT+01:00 > Subject: readBin fails to read large files > > Dear all, > > I am trying to read a large file (~2GB) of unsigned ints into R. Using the command: > > raw<-readBin("file",n=10^8, integer(),endian="little",signed=FALSE) > > It works fine for n=10^8, but fails for n=10^9 (or even at n=6*10^8). My machine$sizeof.long is 8 bit. > I am running R 2.13.1 on a x86_64-apple-darwin9.8.0/x86_64 (64-bit) architecture. > > Thanks for your help > > Florian > > -- > AXA doctoral fellow > Bundy lab - Biomolecular Medicine > Imperial College London > > > > > > -- > AXA doctoral fellow > Bundy lab - Biomolecular Medicine > Imperial College London > > > > > > > ? ? ? ?[[alternative HTML version deleted]] > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. >-- Jim Holtman Data Munger Guru What is the problem that you are trying to solve?
On 01/09/2011 11:13 AM, Benton, Paul wrote:> Posting for a friend >What does "fails" mean, i.e. what is the error message? (You might want to get Florian online here.) Duncan Murdoch> Begin forwarded message: > > From: "Geier, Florian"<florian.geier08 at imperial.ac.uk<mailto:florian.geier08 at imperial.ac.uk>> > Subject: Fwd: readBin fails to read large files > Date: September 1, 2011 4:10:53 PM GMT+01:00 > To: > > > > Begin forwarded message: > > Date: 1 September 2011 16:01:45 GMT+01:00 > Subject: readBin fails to read large files > > Dear all, > > I am trying to read a large file (~2GB) of unsigned ints into R. Using the command: > > raw<-readBin("file",n=10^8, integer(),endian="little",signed=FALSE) > > It works fine for n=10^8, but fails for n=10^9 (or even at n=6*10^8). My machine$sizeof.long is 8 bit. > I am running R 2.13.1 on a x86_64-apple-darwin9.8.0/x86_64 (64-bit) architecture. > > Thanks for your help > > Florian > > -- > AXA doctoral fellow > Bundy lab - Biomolecular Medicine > Imperial College London > > > > > > -- > AXA doctoral fellow > Bundy lab - Biomolecular Medicine > Imperial College London > > > > > > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.
Hi Jim, yes - it definitely is 64 bit. I call it with r64 and .Platform$r_arch [1] "x86_64" It is on a apple snow leopard (10.6.8) with 16 GB of Ram - not windows Florian On 1 Sep 2011, at 16:22, jim holtman wrote:> Are you running a 64-bit version of R? It sounds like your operating > system is not giving you enough memory. It looks like this is not > under Windows in a native mode. > > On Thu, Sep 1, 2011 at 11:13 AM, Benton, Paul > <hpaul.benton08 at imperial.ac.uk> wrote: >> Posting for a friend >> >> Begin forwarded message: >> >> From: "Geier, Florian" <florian.geier08 at imperial.ac.uk<mailto:florian.geier08 at imperial.ac.uk>> >> Subject: Fwd: readBin fails to read large files >> Date: September 1, 2011 4:10:53 PM GMT+01:00 >> To: >> >> >> >> Begin forwarded message: >> >> Date: 1 September 2011 16:01:45 GMT+01:00 >> Subject: readBin fails to read large files >> >> Dear all, >> >> I am trying to read a large file (~2GB) of unsigned ints into R. Using the command: >> >> raw<-readBin("file",n=10^8, integer(),endian="little",signed=FALSE) >> >> It works fine for n=10^8, but fails for n=10^9 (or even at n=6*10^8). My machine$sizeof.long is 8 bit. >> I am running R 2.13.1 on a x86_64-apple-darwin9.8.0/x86_64 (64-bit) architecture. >> >> Thanks for your help >> >> Florian >> >> -- >> AXA doctoral fellow >> Bundy lab - Biomolecular Medicine >> Imperial College London >> >> >> >> >> >> -- >> AXA doctoral fellow >> Bundy lab - Biomolecular Medicine >> Imperial College London >> >> >> >> >> >> >> [[alternative HTML version deleted]] >> >> ______________________________________________ >> R-help at r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> > > > > -- > Jim Holtman > Data Munger Guru > > What is the problem that you are trying to solve? > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.-- AXA doctoral fellow Bundy lab - Biomolecular Medicine Imperial College London
readBin is intended to read a few items at a time, not 10^9. You are probably getting 32-bit integer overflow inside your OS, since the number of bytes you are trying to read in one go exceeds 2GB. Don't do that: read say a million at time. And BTW, if these really are unsigned ints you will get wraparound. On Thu, 1 Sep 2011, Benton, Paul wrote:> Posting for a friend > > Begin forwarded message: > > From: "Geier, Florian" <florian.geier08 at imperial.ac.uk<mailto:florian.geier08 at imperial.ac.uk>> > Subject: Fwd: readBin fails to read large files > Date: September 1, 2011 4:10:53 PM GMT+01:00 > To: > > > > Begin forwarded message: > > Date: 1 September 2011 16:01:45 GMT+01:00 > Subject: readBin fails to read large files > > Dear all, > > I am trying to read a large file (~2GB) of unsigned ints into R. Using the command: > > raw<-readBin("file",n=10^8, integer(),endian="little",signed=FALSE) > > It works fine for n=10^8, but fails for n=10^9 (or even at n=6*10^8). My machine$sizeof.long is 8 bit. > I am running R 2.13.1 on a x86_64-apple-darwin9.8.0/x86_64 (64-bit) architecture. > > Thanks for your help > > Florian > > -- > AXA doctoral fellow > Bundy lab - Biomolecular Medicine > Imperial College London > > > > > > -- > AXA doctoral fellow > Bundy lab - Biomolecular Medicine > Imperial College London > > > > > > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. >-- Brian D. Ripley, ripley at stats.ox.ac.uk Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UK Fax: +44 1865 272595