polzehl@wias-berlin.de
2002-Jun-28 09:53 UTC
Problem in optim(method="L-BFGS-B") (PR#1717)
Full_Name: Jörg Polzehl Version: 1.5.1 OS: Windows 2000 Submission from: (NULL) (193.175.148.198) When calculating MLE's in a variance component model using constrained optimization, i.e. optim(...,method="L-BFGS-B",...) I observed an inproper behaviour in cases where the likelihood function was evalueted at the constraint. Parameters and value of the function at the constraint seem to be returned in this case. I've tried to reproduce the effect in a very simple example: ############################################################################### fn <- function(par,x,y){ ind <- 1:(length(x)/2) alpha <- par[1] beta <- par[2] sigmaq1 <- par[3] sigmaq2 <- par[4] sum((y[ind]-alpha-beta*x[ind])^2/sigmaq1+log(sigmaq1)+(y[-ind]-alpha-beta*x[-ind])^2/sigmaq2+log(sigmaq2)) } set.seed(3) n <- 5 x1 <- runif(n) x2 <- runif(n) y <- c(rnorm(x1,1+x1,1),rnorm(x2,1+x2,5)) x <- c(x1,x2) par <- c(0,0,1,1) z <- optim(par,fn,method="L-BFGS-B",lower=c(-Inf,-Inf,1e-6,1e-6),x=x,y=y,control=list(trace=6,REPORT=1)) #################################################################################### Note that the effect vanishes in case of other constraints for par[3:4]. here is what happens with trace=1 : z <- optim(par,fn,method="L-BFGS-B",lower=c(-Inf,-Inf,1e-6,1e-6),x=x,y=y,control=list(trace=1,REPORT=1)) iter 0 value 121.713156 iter 1 value 98.361657 iter 2 value 66.561121 iter 3 value 57.491023 iter 4 value 48.193824 iter 5 value 42.298821 iter 6 value 37.554272 iter 7 value 33.741241 iter 8 value 31.592504 iter 9 value 28.856846 iter 10 value 27.450574 iter 11 value 4118799.069149 final value 4118799.069149 converged ---------------------------------------------------------------------------------- and the essential part with trace = 6 : 4 variables are free at GCP on iteration 11 LINE SEARCH 1 times; norm of step = 1.19245 X = -3.34057 6.34639 1.18546 15.2136 G = -1.08654 -1.14056 1.42527 -0.307525 Iteration 11 ---------------- CAUCHY entered------------------- There are 1 breakpoints Piece 1 f1, f2 at start point -4.6074e+000 3.1209e+001 Distance to the next break point = 8.3175e-001 Distance to the stationary point = 1.4763e-001 GCP found in this segment Piece 1 f1, f2 at start point -4.6074e+000 3.1209e+001 Distance to the stationary point = 1.4763e-001 Cauchy X = -3.18017 6.51477 0.97505 15.259 ---------------- exit CAUCHY---------------------- 4 variables are free at GCP on iteration 12 # !!!!!!!!!!!!!!!!!!!!!! LINE SEARCH 0 times; norm of step = 2.30637 X = -3.90499 7.26709 1e-006 16.8712 G = 2.86794e+006 2.02055e+006 -4.1147e+009 -0.191468 iterations 12 function evaluations 15 segments explored during Cauchy searches 12 BFGS updates skipped 0 active bounds at final generalized Cauchy point 0 norm of the final projected gradient 4.1147e+009 final function value 4.1188e+006 X = -3.90499 7.26709 1e-006 16.8712 F = 4.1188e+006 final value 4118799.069149 converged See especially the line marked with !!!!!!!!!! Regards, Jörg Polzehl -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-devel mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-devel-request@stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
I can't reproduce this. I get (on Windows R 1.5.1) iter 0 value 121.713156 iter 1 value 98.361657 iter 2 value 66.561121 iter 3 value 57.491023 iter 4 value 48.193824 iter 5 value 42.298821 iter 6 value 37.554272 iter 7 value 33.741241 iter 8 value 31.592504 iter 9 value 28.856846 iter 10 value 27.450574 iter 11 value 26.807136 iter 12 value 26.525653 iter 13 value 25.553618 iter 14 value 25.424473 iter 15 value 25.312266 iter 16 value 25.310750 iter 17 value 25.309939 iter 18 value 25.307804 iter 19 value 25.307030 iter 20 value 25.302144 iter 21 value 25.294243 iter 22 value 25.283499 iter 23 value 25.276034 iter 24 value 25.274281 iter 25 value 25.273023 iter 26 value 25.272970 iter 27 value 25.272845 iter 28 value 25.272737 iter 29 value 25.272729 iter 30 value 25.272662 iter 31 value 25.272652 iter 32 value 25.272652 iter 33 value 25.272652 final value 25.272652 converged However, Linux gives similar problems. The underlying cause is that you are using finite-difference approximations and not scaling your variables as described on the help page. On Fri, 28 Jun 2002 polzehl@wias-berlin.de wrote:> Full_Name: Jörg Polzehl > Version: 1.5.1 > OS: Windows 2000 > Submission from: (NULL) (193.175.148.198) > > > When calculating MLE's in a variance component model using constrained > optimization, i.e. optim(...,method="L-BFGS-B",...) I observed an inproper > behaviour in cases where > the likelihood function was evalueted at the constraint. Parameters and value of > the > function at the constraint seem to be returned in this case. > I've tried to reproduce the effect in a very simple example: > > ############################################################################### > fn <- function(par,x,y){ > ind <- 1:(length(x)/2) > alpha <- par[1] > beta <- par[2] > sigmaq1 <- par[3] > sigmaq2 <- par[4] > sum((y[ind]-alpha-beta*x[ind])^2/sigmaq1+log(sigmaq1)+(y[-ind]-alpha-beta*x[-ind])^2/sigmaq2+log(sigmaq2)) > } > > set.seed(3) > n <- 5 > x1 <- runif(n) > x2 <- runif(n) > y <- c(rnorm(x1,1+x1,1),rnorm(x2,1+x2,5)) > x <- c(x1,x2) > par <- c(0,0,1,1) > z <- optim(par,fn,method="L-BFGS-B",lower=c(-Inf,-Inf,1e-6,1e-6),x=x,y=y,control=list(trace=6,REPORT=1)) > #################################################################################### > > Note that the effect vanishes in case of other constraints for par[3:4]. > > here is what happens with trace=1 : > > z <- optim(par,fn,method="L-BFGS-B",lower=c(-Inf,-Inf,1e-6,1e-6),x=x,y=y,control=list(trace=1,REPORT=1)) > iter 0 value 121.713156 > iter 1 value 98.361657 > iter 2 value 66.561121 > iter 3 value 57.491023 > iter 4 value 48.193824 > iter 5 value 42.298821 > iter 6 value 37.554272 > iter 7 value 33.741241 > iter 8 value 31.592504 > iter 9 value 28.856846 > iter 10 value 27.450574 > iter 11 value 4118799.069149 > final value 4118799.069149 > converged > ---------------------------------------------------------------------------------- > and the essential part with trace = 6 : > > 4 variables are free at GCP on iteration 11 > LINE SEARCH 1 times; norm of step = 1.19245 > X = -3.34057 6.34639 1.18546 15.2136 > G = -1.08654 -1.14056 1.42527 -0.307525 > Iteration 11 > > ---------------- CAUCHY entered------------------- > > There are 1 breakpoints > > Piece 1 f1, f2 at start point -4.6074e+000 3.1209e+001 > Distance to the next break point = 8.3175e-001 > Distance to the stationary point = 1.4763e-001 > > GCP found in this segment > Piece 1 f1, f2 at start point -4.6074e+000 3.1209e+001 > Distance to the stationary point = 1.4763e-001 > Cauchy X = -3.18017 6.51477 0.97505 15.259 > > ---------------- exit CAUCHY---------------------- > > 4 variables are free at GCP on iteration 12 # > !!!!!!!!!!!!!!!!!!!!!! > LINE SEARCH 0 times; norm of step = 2.30637 > X = -3.90499 7.26709 1e-006 16.8712 > G = 2.86794e+006 2.02055e+006 -4.1147e+009 -0.191468 > > iterations 12 > function evaluations 15 > segments explored during Cauchy searches 12 > BFGS updates skipped 0 > active bounds at final generalized Cauchy point 0 > norm of the final projected gradient 4.1147e+009 > final function value 4.1188e+006 > > X = -3.90499 7.26709 1e-006 16.8712 > F = 4.1188e+006 > final value 4118799.069149 > converged > > > See especially the line marked with !!!!!!!!!! > > Regards, > > Jörg Polzehl > > > -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- > r-devel mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html > Send "info", "help", or "[un]subscribe" > (in the "body", not the subject !) To: r-devel-request@stat.math.ethz.ch > _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._ >-- Brian D. Ripley, ripley@stats.ox.ac.uk Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272860 (secr) Oxford OX1 3TG, UK Fax: +44 1865 272595 -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.- r-devel mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html Send "info", "help", or "[un]subscribe" (in the "body", not the subject !) To: r-devel-request@stat.math.ethz.ch _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._