In this email to the R-help list:
https://stat.ethz.ch/pipermail/r-help/2017-June/447474.html
and in this question on Stackoverflow:
https://stackoverflow.com/questions/44486643/nleqslv-memory-use-in-r
Andrew Leach has raised a question about the memory usage of my package nleqslv.
In a model with a loop within a function he has experienced continuously
increasing memory usage
by package nleqslv leading to an out of memory condition.
I have been able to reproduce the continuously increasing memory usage with the
following small example.
<code>
library(nleqslv,lib.loc="../nleqslv.Rcheck")
library(pryr)
dslnex <- function(x) {
y <- numeric(2)
y[1] <- x[1]^2 + x[2]^2 - 2
y[2] <- exp(x[1]-1) + x[2]^3 - 2
y
}
xstart <- c(x1=1.5,x2=2)
nsims <- 10
for(test_iter in seq_len(nsims)){
z <- nleqslv(xstart,dslnex,jacobian=NULL)
print(paste("nleqslv iteration",test_iter,"and memory used
is",mem_used()))
}
memory.profile()
gc()
print(paste("At end memory used is", mem_used()))
</code>
The final output is
<output>
[1] "nleqslv iteration 1 and memory used is 28921288"
[1] "nleqslv iteration 2 and memory used is 29133256"
[1] "nleqslv iteration 3 and memory used is 29132992"
[1] "nleqslv iteration 4 and memory used is 29134712"
[1] "nleqslv iteration 5 and memory used is 29136432"
[1] "nleqslv iteration 6 and memory used is 29138152"
[1] "nleqslv iteration 7 and memory used is 29139872"
[1] "nleqslv iteration 8 and memory used is 29141592"
[1] "nleqslv iteration 9 and memory used is 29143312"
[1] "nleqslv iteration 10 and memory used is
29145032">
> memory.profile()
NULL symbol pairlist closure environment promise
1 8554 194726 4230 1065 6534
language special builtin char logical integer
51374 45 671 9786 8030 37258
double complex character ... any list
2645 1 53795 0 0 18487
expression bytecode externalptr weakref raw S4
1 14662 2233 592 593
1049> gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 416340 22.3 750400 40.1 581181 31.1
Vcells 726127 5.6 1308461 10.0 1191521 9.1> print(paste("At end memory used is", mem_used()))
[1] "At end memory used is 29126472"
</output>
Indeed memory used increases in each pass of the for loop.
I have added these two lines at the top of the code after the library(pryr)
invocation.
library(compiler)
oldJIT <- enableJIT(0)
This resolves the issue in the sense that memory used remains constant after the
first iteration.
Output in the for loop is now:
<output>
[1] "nleqslv iteration 1 and memory used is 24487784"
[1] "nleqslv iteration 2 and memory used is 24495816"
[1] "nleqslv iteration 3 and memory used is 24495816"
[1] "nleqslv iteration 4 and memory used is 24495816"
[1] "nleqslv iteration 5 and memory used is 24495816"
[1] "nleqslv iteration 6 and memory used is 24495816"
[1] "nleqslv iteration 7 and memory used is 24495816"
[1] "nleqslv iteration 8 and memory used is 24495816"
[1] "nleqslv iteration 9 and memory used is 24495816"
[1] "nleqslv iteration 10 and memory used is 24495816"
</output>
My questions are
- is this a bug(let) in the JIT compiler?
- if it isn't what would need to be changed in nleqslv.R in the package
source? (I haven't a clue)
regards
Berend Hasselman
My sessionInfo:
R version 3.4.0 (2017-04-21)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Sierra 10.12.5
Matrix products: default
BLAS:
/Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
LAPACK:
/Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib
locale:
[1] en_IE.UTF-8/en_IE.UTF-8/en_IE.UTF-8/C/en_IE.UTF-8/en_IE.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] compiler_3.4.0
Thanks. This should be resolved in R-devel(r72788) and R-patched (r72789) Best, luke On Mon, 12 Jun 2017, Berend Hasselman wrote:> > > In this email to the R-help list: https://stat.ethz.ch/pipermail/r-help/2017-June/447474.html > and in this question on Stackoverflow: https://stackoverflow.com/questions/44486643/nleqslv-memory-use-in-r > > Andrew Leach has raised a question about the memory usage of my package nleqslv. > In a model with a loop within a function he has experienced continuously increasing memory usage > by package nleqslv leading to an out of memory condition. > > I have been able to reproduce the continuously increasing memory usage with the following small example. > > <code> > library(nleqslv,lib.loc="../nleqslv.Rcheck") > library(pryr) > dslnex <- function(x) { > y <- numeric(2) > y[1] <- x[1]^2 + x[2]^2 - 2 > y[2] <- exp(x[1]-1) + x[2]^3 - 2 > y > } > xstart <- c(x1=1.5,x2=2) > nsims <- 10 > for(test_iter in seq_len(nsims)){ > z <- nleqslv(xstart,dslnex,jacobian=NULL) > print(paste("nleqslv iteration",test_iter,"and memory used is",mem_used())) > } > memory.profile() > gc() > print(paste("At end memory used is", mem_used())) > </code> > > The final output is > > <output> > [1] "nleqslv iteration 1 and memory used is 28921288" > [1] "nleqslv iteration 2 and memory used is 29133256" > [1] "nleqslv iteration 3 and memory used is 29132992" > [1] "nleqslv iteration 4 and memory used is 29134712" > [1] "nleqslv iteration 5 and memory used is 29136432" > [1] "nleqslv iteration 6 and memory used is 29138152" > [1] "nleqslv iteration 7 and memory used is 29139872" > [1] "nleqslv iteration 8 and memory used is 29141592" > [1] "nleqslv iteration 9 and memory used is 29143312" > [1] "nleqslv iteration 10 and memory used is 29145032" >> >> memory.profile() > NULL symbol pairlist closure environment promise > 1 8554 194726 4230 1065 6534 > language special builtin char logical integer > 51374 45 671 9786 8030 37258 > double complex character ... any list > 2645 1 53795 0 0 18487 > expression bytecode externalptr weakref raw S4 > 1 14662 2233 592 593 1049 >> gc() > used (Mb) gc trigger (Mb) max used (Mb) > Ncells 416340 22.3 750400 40.1 581181 31.1 > Vcells 726127 5.6 1308461 10.0 1191521 9.1 >> print(paste("At end memory used is", mem_used())) > [1] "At end memory used is 29126472" > </output> > > Indeed memory used increases in each pass of the for loop. > > I have added these two lines at the top of the code after the library(pryr) invocation. > > library(compiler) > oldJIT <- enableJIT(0) > > This resolves the issue in the sense that memory used remains constant after the first iteration. > Output in the for loop is now: > <output> > [1] "nleqslv iteration 1 and memory used is 24487784" > [1] "nleqslv iteration 2 and memory used is 24495816" > [1] "nleqslv iteration 3 and memory used is 24495816" > [1] "nleqslv iteration 4 and memory used is 24495816" > [1] "nleqslv iteration 5 and memory used is 24495816" > [1] "nleqslv iteration 6 and memory used is 24495816" > [1] "nleqslv iteration 7 and memory used is 24495816" > [1] "nleqslv iteration 8 and memory used is 24495816" > [1] "nleqslv iteration 9 and memory used is 24495816" > [1] "nleqslv iteration 10 and memory used is 24495816" > </output> > > My questions are > > - is this a bug(let) in the JIT compiler? > - if it isn't what would need to be changed in nleqslv.R in the package source? (I haven't a clue) > > regards > > Berend Hasselman > > My sessionInfo: > > R version 3.4.0 (2017-04-21) > Platform: x86_64-apple-darwin15.6.0 (64-bit) > Running under: macOS Sierra 10.12.5 > > Matrix products: default > BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib > LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib > > locale: > [1] en_IE.UTF-8/en_IE.UTF-8/en_IE.UTF-8/C/en_IE.UTF-8/en_IE.UTF-8 > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > loaded via a namespace (and not attached): > [1] compiler_3.4.0 > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel >-- Luke Tierney Ralph E. Wareham Professor of Mathematical Sciences University of Iowa Phone: 319-335-3386 Department of Statistics and Fax: 319-335-3017 Actuarial Science 241 Schaeffer Hall email: luke-tierney at uiowa.edu Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu
> On 13 Jun 2017, at 22:05, luke-tierney at uiowa.edu wrote: > > Thanks. This should be resolved in R-devel(r72788) and R-patched > (r72789) >Also thanks. It is resolved in R-patched. Berend> Best, > > luke > > On Mon, 12 Jun 2017, Berend Hasselman wrote: > >> >> >> In this email to the R-help list: https://stat.ethz.ch/pipermail/r-help/2017-June/447474.html >> and in this question on Stackoverflow: https://stackoverflow.com/questions/44486643/nleqslv-memory-use-in-r >> >> Andrew Leach has raised a question about the memory usage of my package nleqslv. >> In a model with a loop within a function he has experienced continuously increasing memory usage >> by package nleqslv leading to an out of memory condition. >> >> I have been able to reproduce the continuously increasing memory usage with the following small example. >> >> <code> >> library(nleqslv,lib.loc="../nleqslv.Rcheck") >> library(pryr) >> dslnex <- function(x) { >> y <- numeric(2) >> y[1] <- x[1]^2 + x[2]^2 - 2 >> y[2] <- exp(x[1]-1) + x[2]^3 - 2 >> y >> } >> xstart <- c(x1=1.5,x2=2) >> nsims <- 10 >> for(test_iter in seq_len(nsims)){ >> z <- nleqslv(xstart,dslnex,jacobian=NULL) >> print(paste("nleqslv iteration",test_iter,"and memory used is",mem_used())) >> } >> memory.profile() >> gc() >> print(paste("At end memory used is", mem_used())) >> </code> >> >> The final output is >> >> <output> >> [1] "nleqslv iteration 1 and memory used is 28921288" >> [1] "nleqslv iteration 2 and memory used is 29133256" >> [1] "nleqslv iteration 3 and memory used is 29132992" >> [1] "nleqslv iteration 4 and memory used is 29134712" >> [1] "nleqslv iteration 5 and memory used is 29136432" >> [1] "nleqslv iteration 6 and memory used is 29138152" >> [1] "nleqslv iteration 7 and memory used is 29139872" >> [1] "nleqslv iteration 8 and memory used is 29141592" >> [1] "nleqslv iteration 9 and memory used is 29143312" >> [1] "nleqslv iteration 10 and memory used is 29145032" >>> >>> memory.profile() >> NULL symbol pairlist closure environment promise >> 1 8554 194726 4230 1065 6534 >> language special builtin char logical integer >> 51374 45 671 9786 8030 37258 >> double complex character ... any list >> 2645 1 53795 0 0 18487 >> expression bytecode externalptr weakref raw S4 >> 1 14662 2233 592 593 1049 >>> gc() >> used (Mb) gc trigger (Mb) max used (Mb) >> Ncells 416340 22.3 750400 40.1 581181 31.1 >> Vcells 726127 5.6 1308461 10.0 1191521 9.1 >>> print(paste("At end memory used is", mem_used())) >> [1] "At end memory used is 29126472" >> </output> >> >> Indeed memory used increases in each pass of the for loop. >> >> I have added these two lines at the top of the code after the library(pryr) invocation. >> >> library(compiler) >> oldJIT <- enableJIT(0) >> >> This resolves the issue in the sense that memory used remains constant after the first iteration. >> Output in the for loop is now: >> <output> >> [1] "nleqslv iteration 1 and memory used is 24487784" >> [1] "nleqslv iteration 2 and memory used is 24495816" >> [1] "nleqslv iteration 3 and memory used is 24495816" >> [1] "nleqslv iteration 4 and memory used is 24495816" >> [1] "nleqslv iteration 5 and memory used is 24495816" >> [1] "nleqslv iteration 6 and memory used is 24495816" >> [1] "nleqslv iteration 7 and memory used is 24495816" >> [1] "nleqslv iteration 8 and memory used is 24495816" >> [1] "nleqslv iteration 9 and memory used is 24495816" >> [1] "nleqslv iteration 10 and memory used is 24495816" >> </output> >> >> My questions are >> >> - is this a bug(let) in the JIT compiler? >> - if it isn't what would need to be changed in nleqslv.R in the package source? (I haven't a clue) >> >> regards >> >> Berend Hasselman >> >> My sessionInfo: >> >> R version 3.4.0 (2017-04-21) >> Platform: x86_64-apple-darwin15.6.0 (64-bit) >> Running under: macOS Sierra 10.12.5 >> >> Matrix products: default >> BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib >> LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib >> >> locale: >> [1] en_IE.UTF-8/en_IE.UTF-8/en_IE.UTF-8/C/en_IE.UTF-8/en_IE.UTF-8 >> >> attached base packages: >> [1] stats graphics grDevices utils datasets methods base >> >> loaded via a namespace (and not attached): >> [1] compiler_3.4.0 >> >> ______________________________________________ >> R-devel at r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-devel >> > > -- > Luke Tierney > Ralph E. Wareham Professor of Mathematical Sciences > University of Iowa Phone: 319-335-3386 > Department of Statistics and Fax: 319-335-3017 > Actuarial Science > 241 Schaeffer Hall email: luke-tierney at uiowa.edu > Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu