I have a data file called inputdata.csv that looks something like this"
ID Year Result Month Date
1 7174 1954 10 3 540301
2 7174 1954 4 3 540322
3 20924 1967 4 2 670223
4 20924 1967 -7 5 670518
5 20924 1967 -3 7 670706
...
67209 ...
i.e., it goes on for 67209 rows (~2 Mb file). When I run the following
bootstrap session I get the indicated error:
>
> library(boot)
> setwd("C:/Documents and Settings/Tom/Desktop")
>
> data.in <- read.csv("inputdata.csv",header=T,as.is=T)
>
> per95 <- function( annual.data, b.index) {
+ sample.data <- annual.data[b.index,]
+ return(quantile(sample.data$Result,probs=c(0.95))) }>
> m <- 10000
> for (i in 1:39) {
+ annual.data <- data.in[data.in$Year == (i+1949),]
+ B <- boot(data=annual.data,statistic=per95,R=m)
+ print(i)
+ print(memory.size())
+ }
[1] 1
[1] 20.26163
[1] 2
[1] 61.6352
[1] 3
[1] 134.4187
[1] 4
[1] 149.4704
[1] 5
[1] 290.3090
[1] 6
[1] 376.7017
[1] 7
[1] 435.7683
[1] 8
[1] 463.7404
[1] 9
[1] 497.7946
Error: cannot allocate vector of size 568.8 Mb>
I am running this on a Windows XP Pro machine with 4 Gb of memory. The same
problem occurs when the code is executed on the same box running Ubuntu
8.04. Does anyone see any obvious reason why this should run out of memory?
I would be happy to email the data file to anyone who cares to try it on
their computer.
Tom
--
View this message in context:
http://www.nabble.com/Memory-Problems-with-a-Simple-Bootstrap-tp18777897p18777897.html
Sent from the R help mailing list archive at Nabble.com.
Use gc() in the loop to possibly free up any fragmented memory. You might also print out the size of B (object.size(B)) since that appears to be the only variable in your loop that might be growing. On Fri, Aug 1, 2008 at 12:09 PM, Tom La Bone <booboo at gforcecable.com> wrote:> > > I have a data file called inputdata.csv that looks something like this" > > ID Year Result Month Date > 1 7174 1954 10 3 540301 > 2 7174 1954 4 3 540322 > 3 20924 1967 4 2 670223 > 4 20924 1967 -7 5 670518 > 5 20924 1967 -3 7 670706 > ... > 67209 ... > > i.e., it goes on for 67209 rows (~2 Mb file). When I run the following > bootstrap session I get the indicated error: > >> >> library(boot) >> setwd("C:/Documents and Settings/Tom/Desktop") >> >> data.in <- read.csv("inputdata.csv",header=T,as.is=T) >> >> per95 <- function( annual.data, b.index) { > + sample.data <- annual.data[b.index,] > + return(quantile(sample.data$Result,probs=c(0.95))) } >> >> m <- 10000 >> for (i in 1:39) { > + annual.data <- data.in[data.in$Year == (i+1949),] > + B <- boot(data=annual.data,statistic=per95,R=m) > + print(i) > + print(memory.size()) > + } > [1] 1 > [1] 20.26163 > [1] 2 > [1] 61.6352 > [1] 3 > [1] 134.4187 > [1] 4 > [1] 149.4704 > [1] 5 > [1] 290.3090 > [1] 6 > [1] 376.7017 > [1] 7 > [1] 435.7683 > [1] 8 > [1] 463.7404 > [1] 9 > [1] 497.7946 > Error: cannot allocate vector of size 568.8 Mb >> > > I am running this on a Windows XP Pro machine with 4 Gb of memory. The same > problem occurs when the code is executed on the same box running Ubuntu > 8.04. Does anyone see any obvious reason why this should run out of memory? > I would be happy to email the data file to anyone who cares to try it on > their computer. > > Tom > > > > > > -- > View this message in context: http://www.nabble.com/Memory-Problems-with-a-Simple-Bootstrap-tp18777897p18777897.html > Sent from the R help mailing list archive at Nabble.com. > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. >-- Jim Holtman Cincinnati, OH +1 513 646 9390 What is the problem that you are trying to solve?
Same problem. The Windows Task Manager indicated that Rgui.exe was using 1,249,722 K of memory when the error occurred. This is R 2.7.1 by the way.> library(boot) > setwd("C:/Documents and Settings/Tom/Desktop") > > data.in <- read.csv("inputdata.csv",header=T,as.is=T) > > per95 <- function( annual.data, b.index) {+ sample.data <- annual.data[b.index,] + return(quantile(sample.data$Result,probs=c(0.95))) }> > m <- 10000 > for (i in 1:39) {+ annual.data <- data.in[data.in$Year == (i+1949),] + B <- boot(data=annual.data,statistic=per95,R=m) + gc() + print(i) + print(object.size(B)) + print(memory.size()) + } [1] 1 [1] 90352 [1] 12.35335 [1] 2 [1] 111032 [1] 12.39024 [1] 3 [1] 155544 [1] 12.48451 [1] 4 [1] 159064 [1] 11.10526 [1] 5 [1] 243456 [1] 11.23505 [1] 6 [1] 280592 [1] 12.74642 [1] 7 [1] 302416 [1] 11.33087 [1] 8 [1] 319752 [1] 12.84377 [1] 9 [1] 351448 [1] 11.42264 Error: cannot allocate vector of size 284.4 Mb> >jholtman wrote:> > Use gc() in the loop to possibly free up any fragmented memory. You > might also print out the size of B (object.size(B)) since that appears > to be the only variable in your loop that might be growing. > > On Fri, Aug 1, 2008 at 12:09 PM, Tom La Bone <booboo at gforcecable.com> > wrote: >> >> >> I have a data file called inputdata.csv that looks something like this" >> >> ID Year Result Month Date >> 1 7174 1954 10 3 540301 >> 2 7174 1954 4 3 540322 >> 3 20924 1967 4 2 670223 >> 4 20924 1967 -7 5 670518 >> 5 20924 1967 -3 7 670706 >> ... >> 67209 ... >> >> i.e., it goes on for 67209 rows (~2 Mb file). When I run the following >> bootstrap session I get the indicated error: >> >>> >>> library(boot) >>> setwd("C:/Documents and Settings/Tom/Desktop") >>> >>> data.in <- read.csv("inputdata.csv",header=T,as.is=T) >>> >>> per95 <- function( annual.data, b.index) { >> + sample.data <- annual.data[b.index,] >> + return(quantile(sample.data$Result,probs=c(0.95))) } >>> >>> m <- 10000 >>> for (i in 1:39) { >> + annual.data <- data.in[data.in$Year == (i+1949),] >> + B <- boot(data=annual.data,statistic=per95,R=m) >> + print(i) >> + print(memory.size()) >> + } >> [1] 1 >> [1] 20.26163 >> [1] 2 >> [1] 61.6352 >> [1] 3 >> [1] 134.4187 >> [1] 4 >> [1] 149.4704 >> [1] 5 >> [1] 290.3090 >> [1] 6 >> [1] 376.7017 >> [1] 7 >> [1] 435.7683 >> [1] 8 >> [1] 463.7404 >> [1] 9 >> [1] 497.7946 >> Error: cannot allocate vector of size 568.8 Mb >>> >> >> I am running this on a Windows XP Pro machine with 4 Gb of memory. The >> same >> problem occurs when the code is executed on the same box running Ubuntu >> 8.04. Does anyone see any obvious reason why this should run out of >> memory? >> I would be happy to email the data file to anyone who cares to try it on >> their computer. >> >> Tom >> >> >> >> >> >> -- >> View this message in context: >> http://www.nabble.com/Memory-Problems-with-a-Simple-Bootstrap-tp18777897p18777897.html >> Sent from the R help mailing list archive at Nabble.com. >> >> ______________________________________________ >> R-help at r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide >> http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> > > > > -- > Jim Holtman > Cincinnati, OH > +1 513 646 9390 > > What is the problem that you are trying to solve? > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > >-- View this message in context: http://www.nabble.com/Memory-Problems-with-a-Simple-Bootstrap-tp18777897p18779433.html Sent from the R help mailing list archive at Nabble.com.
It seems like the objects are reasonable size and the memory size also seems reasonable. That is what I usually go by to see if there are large objects in my memory. If it was showing that R had 1.2GB of memory allocated to it, I wonder if there might be a memory leak somewhere. On Fri, Aug 1, 2008 at 1:36 PM, Tom La Bone <booboo at gforcecable.com> wrote:> > Same problem. The Windows Task Manager indicated that Rgui.exe was using > 1,249,722 K of memory when the error occurred. This is R 2.7.1 by the way. > >> library(boot) >> setwd("C:/Documents and Settings/Tom/Desktop") >> >> data.in <- read.csv("inputdata.csv",header=T,as.is=T) >> >> per95 <- function( annual.data, b.index) { > + sample.data <- annual.data[b.index,] > + return(quantile(sample.data$Result,probs=c(0.95))) } >> >> m <- 10000 >> for (i in 1:39) { > + annual.data <- data.in[data.in$Year == (i+1949),] > + B <- boot(data=annual.data,statistic=per95,R=m) > + gc() > + print(i) > + print(object.size(B)) > + print(memory.size()) > + } > [1] 1 > [1] 90352 > [1] 12.35335 > [1] 2 > [1] 111032 > [1] 12.39024 > [1] 3 > [1] 155544 > [1] 12.48451 > [1] 4 > [1] 159064 > [1] 11.10526 > [1] 5 > [1] 243456 > [1] 11.23505 > [1] 6 > [1] 280592 > [1] 12.74642 > [1] 7 > [1] 302416 > [1] 11.33087 > [1] 8 > [1] 319752 > [1] 12.84377 > [1] 9 > [1] 351448 > [1] 11.42264 > Error: cannot allocate vector of size 284.4 Mb >> >> > > > > jholtman wrote: >> >> Use gc() in the loop to possibly free up any fragmented memory. You >> might also print out the size of B (object.size(B)) since that appears >> to be the only variable in your loop that might be growing. >> >> On Fri, Aug 1, 2008 at 12:09 PM, Tom La Bone <booboo at gforcecable.com> >> wrote: >>> >>> >>> I have a data file called inputdata.csv that looks something like this" >>> >>> ID Year Result Month Date >>> 1 7174 1954 10 3 540301 >>> 2 7174 1954 4 3 540322 >>> 3 20924 1967 4 2 670223 >>> 4 20924 1967 -7 5 670518 >>> 5 20924 1967 -3 7 670706 >>> ... >>> 67209 ... >>> >>> i.e., it goes on for 67209 rows (~2 Mb file). When I run the following >>> bootstrap session I get the indicated error: >>> >>>> >>>> library(boot) >>>> setwd("C:/Documents and Settings/Tom/Desktop") >>>> >>>> data.in <- read.csv("inputdata.csv",header=T,as.is=T) >>>> >>>> per95 <- function( annual.data, b.index) { >>> + sample.data <- annual.data[b.index,] >>> + return(quantile(sample.data$Result,probs=c(0.95))) } >>>> >>>> m <- 10000 >>>> for (i in 1:39) { >>> + annual.data <- data.in[data.in$Year == (i+1949),] >>> + B <- boot(data=annual.data,statistic=per95,R=m) >>> + print(i) >>> + print(memory.size()) >>> + } >>> [1] 1 >>> [1] 20.26163 >>> [1] 2 >>> [1] 61.6352 >>> [1] 3 >>> [1] 134.4187 >>> [1] 4 >>> [1] 149.4704 >>> [1] 5 >>> [1] 290.3090 >>> [1] 6 >>> [1] 376.7017 >>> [1] 7 >>> [1] 435.7683 >>> [1] 8 >>> [1] 463.7404 >>> [1] 9 >>> [1] 497.7946 >>> Error: cannot allocate vector of size 568.8 Mb >>>> >>> >>> I am running this on a Windows XP Pro machine with 4 Gb of memory. The >>> same >>> problem occurs when the code is executed on the same box running Ubuntu >>> 8.04. Does anyone see any obvious reason why this should run out of >>> memory? >>> I would be happy to email the data file to anyone who cares to try it on >>> their computer. >>> >>> Tom >>> >>> >>> >>> >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/Memory-Problems-with-a-Simple-Bootstrap-tp18777897p18777897.html >>> Sent from the R help mailing list archive at Nabble.com. >>> >>> ______________________________________________ >>> R-help at r-project.org mailing list >>> https://stat.ethz.ch/mailman/listinfo/r-help >>> PLEASE do read the posting guide >>> http://www.R-project.org/posting-guide.html >>> and provide commented, minimal, self-contained, reproducible code. >>> >> >> >> >> -- >> Jim Holtman >> Cincinnati, OH >> +1 513 646 9390 >> >> What is the problem that you are trying to solve? >> >> ______________________________________________ >> R-help at r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide >> http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> >> > > -- > View this message in context: http://www.nabble.com/Memory-Problems-with-a-Simple-Bootstrap-tp18777897p18779433.html > Sent from the R help mailing list archive at Nabble.com. > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. >-- Jim Holtman Cincinnati, OH +1 513 646 9390 What is the problem that you are trying to solve?
Here it is with the gc() in a print statement. Tom> > library(boot) > setwd("C:/Documents and Settings/Tom/Desktop") > > data.in <- read.csv("inputdata.csv",header=T,as.is=T) > > per95 <- function( annual.data, b.index) {+ sample.data <- annual.data[b.index,] + return(quantile(sample.data$Result,probs=c(0.95))) }> > m <- 10000 > for (i in 1:39) {+ annual.data <- data.in[data.in$Year == (i+1949),] + B <- boot(data=annual.data,statistic=per95,R=m) + print(i) + print(gc()) + print(object.size(B)) + print(memory.size()) + } [1] 1 used (Mb) gc trigger (Mb) max used (Mb) Ncells 145517 3.9 350000 9.4 350000 9.4 Vcells 304805 2.4 2602013 19.9 2841664 21.7 [1] 90352 [1] 12.35812 [1] 2 used (Mb) gc trigger (Mb) max used (Mb) Ncells 145540 3.9 350000 9.4 350000 9.4 Vcells 309041 2.4 12977679 99.1 15259760 116.5 [1] 111032 [1] 12.39814 [1] 3 used (Mb) gc trigger (Mb) max used (Mb) Ncells 145540 3.9 350000 9.4 350000 9.4 Vcells 318147 2.5 35277418 269.2 41833896 319.2 [1] 155544 [1] 12.49432 [1] 4 used (Mb) gc trigger (Mb) max used (Mb) Ncells 145540 3.9 350000 9.4 350000 9.4 Vcells 318867 2.5 37046714 282.7 43935337 335.3 [1] 159064 [1] 11.10362 [1] 5 used (Mb) gc trigger (Mb) max used (Mb) Ncells 145540 3.9 350000 9.4 350000 9.4 Vcells 336129 2.6 79348305 605.4 94192718 718.7 [1] 243456 [1] 11.2296 [1] 6 used (Mb) gc trigger (Mb) max used (Mb) Ncells 145540 3.9 350000 9.4 350000 9.4 Vcells 343725 2.7 97971904 747.5 116530118 889.1 [1] 280592 [1] 12.75431 [1] 7 used (Mb) gc trigger (Mb) max used (Mb) Ncells 145540 3.9 350000 9.4 350000 9.4 Vcells 348189 2.7 108915204 831.0 129493067 988.0 [1] 302416 [1] 11.32924 [1] 8 used (Mb) gc trigger (Mb) max used (Mb) Ncells 145540 3.9 350000 9.4 350000 9.4 Vcells 351735 2.7 117607222 897.3 139706454 1065.9 [1] 319752 [1] 12.85929 [1] 9 used (Mb) gc trigger (Mb) max used (Mb) Ncells 145540 3.9 350000 9.4 350000 9.4 Vcells 358217 2.8 133510676 1018.7 158462984 1209.0 [1] 351448 [1] 11.40765 Error: cannot allocate vector of size 284.4 Mb> > >jholtman wrote:> > It seems like the objects are reasonable size and the memory size also > seems reasonable. That is what I usually go by to see if there are > large objects in my memory. If it was showing that R had 1.2GB of > memory allocated to it, I wonder if there might be a memory leak > somewhere. > > On Fri, Aug 1, 2008 at 1:36 PM, Tom La Bone <booboo at gforcecable.com> > wrote: >> >> Same problem. The Windows Task Manager indicated that Rgui.exe was using >> 1,249,722 K of memory when the error occurred. This is R 2.7.1 by the >> way. >> >>> library(boot) >>> setwd("C:/Documents and Settings/Tom/Desktop") >>> >>> data.in <- read.csv("inputdata.csv",header=T,as.is=T) >>> >>> per95 <- function( annual.data, b.index) { >> + sample.data <- annual.data[b.index,] >> + return(quantile(sample.data$Result,probs=c(0.95))) } >>> >>> m <- 10000 >>> for (i in 1:39) { >> + annual.data <- data.in[data.in$Year == (i+1949),] >> + B <- boot(data=annual.data,statistic=per95,R=m) >> + gc() >> + print(i) >> + print(object.size(B)) >> + print(memory.size()) >> + } >> [1] 1 >> [1] 90352 >> [1] 12.35335 >> [1] 2 >> [1] 111032 >> [1] 12.39024 >> [1] 3 >> [1] 155544 >> [1] 12.48451 >> [1] 4 >> [1] 159064 >> [1] 11.10526 >> [1] 5 >> [1] 243456 >> [1] 11.23505 >> [1] 6 >> [1] 280592 >> [1] 12.74642 >> [1] 7 >> [1] 302416 >> [1] 11.33087 >> [1] 8 >> [1] 319752 >> [1] 12.84377 >> [1] 9 >> [1] 351448 >> [1] 11.42264 >> Error: cannot allocate vector of size 284.4 Mb >>> >>> >> >> >> >> jholtman wrote: >>> >>> Use gc() in the loop to possibly free up any fragmented memory. You >>> might also print out the size of B (object.size(B)) since that appears >>> to be the only variable in your loop that might be growing. >>> >>> On Fri, Aug 1, 2008 at 12:09 PM, Tom La Bone <booboo at gforcecable.com> >>> wrote: >>>> >>>> >>>> I have a data file called inputdata.csv that looks something like this" >>>> >>>> ID Year Result Month Date >>>> 1 7174 1954 10 3 540301 >>>> 2 7174 1954 4 3 540322 >>>> 3 20924 1967 4 2 670223 >>>> 4 20924 1967 -7 5 670518 >>>> 5 20924 1967 -3 7 670706 >>>> ... >>>> 67209 ... >>>> >>>> i.e., it goes on for 67209 rows (~2 Mb file). When I run the following >>>> bootstrap session I get the indicated error: >>>> >>>>> >>>>> library(boot) >>>>> setwd("C:/Documents and Settings/Tom/Desktop") >>>>> >>>>> data.in <- read.csv("inputdata.csv",header=T,as.is=T) >>>>> >>>>> per95 <- function( annual.data, b.index) { >>>> + sample.data <- annual.data[b.index,] >>>> + return(quantile(sample.data$Result,probs=c(0.95))) } >>>>> >>>>> m <- 10000 >>>>> for (i in 1:39) { >>>> + annual.data <- data.in[data.in$Year == (i+1949),] >>>> + B <- boot(data=annual.data,statistic=per95,R=m) >>>> + print(i) >>>> + print(memory.size()) >>>> + } >>>> [1] 1 >>>> [1] 20.26163 >>>> [1] 2 >>>> [1] 61.6352 >>>> [1] 3 >>>> [1] 134.4187 >>>> [1] 4 >>>> [1] 149.4704 >>>> [1] 5 >>>> [1] 290.3090 >>>> [1] 6 >>>> [1] 376.7017 >>>> [1] 7 >>>> [1] 435.7683 >>>> [1] 8 >>>> [1] 463.7404 >>>> [1] 9 >>>> [1] 497.7946 >>>> Error: cannot allocate vector of size 568.8 Mb >>>>> >>>> >>>> I am running this on a Windows XP Pro machine with 4 Gb of memory. The >>>> same >>>> problem occurs when the code is executed on the same box running Ubuntu >>>> 8.04. Does anyone see any obvious reason why this should run out of >>>> memory? >>>> I would be happy to email the data file to anyone who cares to try it >>>> on >>>> their computer. >>>> >>>> Tom >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://www.nabble.com/Memory-Problems-with-a-Simple-Bootstrap-tp18777897p18777897.html >>>> Sent from the R help mailing list archive at Nabble.com. >>>> >>>> ______________________________________________ >>>> R-help at r-project.org mailing list >>>> https://stat.ethz.ch/mailman/listinfo/r-help >>>> PLEASE do read the posting guide >>>> http://www.R-project.org/posting-guide.html >>>> and provide commented, minimal, self-contained, reproducible code. >>>> >>> >>> >>> >>> -- >>> Jim Holtman >>> Cincinnati, OH >>> +1 513 646 9390 >>> >>> What is the problem that you are trying to solve? >>> >>> ______________________________________________ >>> R-help at r-project.org mailing list >>> https://stat.ethz.ch/mailman/listinfo/r-help >>> PLEASE do read the posting guide >>> http://www.R-project.org/posting-guide.html >>> and provide commented, minimal, self-contained, reproducible code. >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/Memory-Problems-with-a-Simple-Bootstrap-tp18777897p18779433.html >> Sent from the R help mailing list archive at Nabble.com. >> >> ______________________________________________ >> R-help at r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide >> http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> > > > > -- > Jim Holtman > Cincinnati, OH > +1 513 646 9390 > > What is the problem that you are trying to solve? > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > >-- View this message in context: http://www.nabble.com/Memory-Problems-with-a-Simple-Bootstrap-tp18777897p18781306.html Sent from the R help mailing list archive at Nabble.com.