R help - Feb 2011 - BFGS versus L-BFGS-B

Hi all,

I'm trying to figure out the effective differences between BFGS and L-BFGS-B
are, besides the obvious that L-BFGS-B should be using a lot less memory,
and the user can provide box constraints.

1) Why would you ever want to use BFGS, if L-BFGS-B does the same thing but
use less memory?

2) If i'm optimizing with respect to a variable x that must be non-negative,
a common approach is to do a change of variables x = exp(y), and optimize
unconstrained with respect to y.  Is optimization using box constraints on
x, likely to produce as good a result as unconstrained optimization on y?

- Brian.

	[[alternative HTML version deleted]]

Brian Tsai <btsai00 <at> gmail.com> writes:
> 
> Hi all,
> 
> I'm trying to figure out the effective differences between BFGS and
L-BFGS-B
> are, besides the obvious that L-BFGS-B should be using a lot less memory,
> and the user can provide box constraints.
> 
> 1) Why would you ever want to use BFGS, if L-BFGS-B does the same thing but
> use less memory?
  L-BFGS-B is a bit more finicky: for example, it does not allow
non-finite (infinite or NA) return values from the objective function,
while BFGS does (although neither does during the initial function evaluation).
I don't know offhand of other differences, although speed may differ.
> 2) If i'm optimizing with respect to a variable x that must be
non-negative,
> a common approach is to do a change of variables x = exp(y), and optimize
> unconstrained with respect to y.  Is optimization using box constraints on
> x, likely to produce as good a result as unconstrained optimization on y?
  It depends.  If the optimal solution is on the boundary (i.e. x=0)
then optimization on the transformed variable (I think you mean y=exp(x)
above?) will work very badly. On the other hand, if the solution is in 
the interior then transforming sometimes works even better -- for example,
the goodness-of-fit surface may be closer to quadratic (which sometimes
has advantages in terms of inference) with the transformed than the
untransformed parameter.

  Ben Bolker

There are considerable differences between the algorithms. And BFGS is an
unfortunate
nomenclature, since there are so many variants that are VERY different. It was
called
"variable metric" in my book from which the code was derived, and that
code was from Roger
Fletcher's Fortran VM code based on his 1970 paper. L-BFGS-B is a later and
more
complicated algorithm with some pretty nice properties. The code is much larger.

Re: "less memory" -- this will depend on the number of parameters, but
to my knowledge
there are no good benchmark studies of memory and performance. Perhaps someone
wants to
propose one for Google Summer of Code (see
http://rwiki.sciviews.org/doku.php?id=developers:projects:gsoc2011
).

The optimx package can call Rvmmin which has box constraints (also Rcgmin that
is intended
for very low memory). Also several other methods with box constraints, including
L-BFGS-B.
Worth a try if you are seeking a method for multiple "production"
runs. Unfortunately, we
seem to have some CRAN check errors on Solaris and some old releases --
platforms I do not
have -- so it may be a few days or more until we sort out the issues, which seem
to be
related to alignment of the underlying packages for which optimx is a wrapper.

Use of transformation can be very effective. But again, I don't think there
are good
studies on whether use of box constraints or transformations is
"better" and when. Another
project, which I have made some tentative beginings to carry out. Collaborations
welcome.

Best,

JN


On 02/25/2011 06:00 AM, r-help-request at r-project.org
wrote:
> Message: 86
> Date: Fri, 25 Feb 2011 00:11:59 -0500
> From: Brian Tsai <btsai00 at gmail.com>
> To: r-help at r-project.org
> Subject: [R] BFGS versus L-BFGS-B
> Message-ID:
> 	<AANLkTimSzVkJbUhV-bBR1easPX9oOTJxQCUjGUjR5Lo6 at mail.gmail.com>
> Content-Type: text/plain
> 
> Hi all,
> 
> I'm trying to figure out the effective differences between BFGS and
L-BFGS-B
> are, besides the obvious that L-BFGS-B should be using a lot less memory,
> and the user can provide box constraints.
> 
> 1) Why would you ever want to use BFGS, if L-BFGS-B does the same thing but
> use less memory?
> 
> 2) If i'm optimizing with respect to a variable x that must be
non-negative,
> a common approach is to do a change of variables x = exp(y), and optimize
> unconstrained with respect to y.  Is optimization using box constraints on
> x, likely to produce as good a result as unconstrained optimization on y?
> 
> - Brian.
> 
> 	[[alternative HTML version deleted]]
> 
> 
>