R help - Nov 2007 - How is the Gauss-Newton method compared to Levenberg-Marquardt for curve-fitting?

Hi,

It seems to me that the most suitable method in R for curve-fitting is the use
of nls, which uses a Gauss-Newton (GN) algorithm, while the use of the
Levenberg-Marquardt (LM) algorithm does not seem to be very stressed in R.
According to this [1] by Ripley, 'Levenberg-Marquardt is hardly competitive
these days' which could imply the low emphasize on LM in R.

The position of LM is, to some extend, confusing. Bonnans et al [2] introduce
the trust-region-based method of LM like this:

'This chapter is mostly devoted to methods which, although less
"universal"
than the preceding, are useful in a good number of cases. The frst one
(trust-region)
is actually extremely important, and might supersede line-searches,
sooner or later.'

The above should demonstrate the contradiction.

Since some R developers are indeed the pioneers in the optimisation theory, I
would like to ask for references involving profiling of various methods,
including more modern techniques, with an application in general model-fitting.




[1] http://tolstoy.newcastle.edu.au/R/help/00b/2492.html

[2] Numerical Optimization, 2nd ed
_________________________________________________________________
100?s of Music vouchers to be won with MSN Music

Almost every problem I have seen in the context of fitting mixtures of
exponentials/Gaussians has near-equal performance under LM or GN in terms
of iterations/evaluations to convergence and the SSE of the solution
found.

(However, very recently I have encountered a problem with a large number
of nonlinear parameters (~100) for which LM finds a better optimum than
GN; I am in the process of looking into why this is the case.)

Regarding profiling: I have never looked for such literature; if you find
it, realize that the results may not generalize to the SSE surface of the
residual functions you will consider.  Therefore you might be better off
doing the profiling yourself for the residual functions that are typical
in your problem domain (you can use the R package minpack.lm for LM).

For general discussion of LM and GN, standard referencs are Bates and
Watts, "Nonlinear regression analysis and its applications", and Seber
and
Wild, "Nonlinear regression".

I believe that one of the main reasons that GN gets more use than LM by R
users is that the former is in base R, and for many simple problems,
writing the function to minimize using a formula is handy, which only nls,
but not minpack.lm, allows.  I don't think it's the case that LM is
hardly
"competitive" in general.

On Tue, 20 Nov 2007, Ali - wrote:
>
> Hi,
>
> It seems to me that the most suitable method in R for curve-fitting is
> the use of nls, which uses a Gauss-Newton (GN) algorithm, while the use
> of the Levenberg-Marquardt (LM) algorithm does not seem to be very
> stressed in R. According to this [1] by Ripley, 'Levenberg-Marquardt is
> hardly competitive these days' which could imply the low emphasize on
LM
> in R.
>
> The position of LM is, to some extend, confusing. Bonnans et al [2]
> introduce the trust-region-based method of LM like this:
>
> 'This chapter is mostly devoted to methods which, although less
> "universal" than the preceding, are useful in a good number of
cases.
> The frst one (trust-region) is actually extremely important, and might
> supersede line-searches, sooner or later.'
>
> The above should demonstrate the contradiction.
>
> Since some R developers are indeed the pioneers in the optimisation
> theory, I would like to ask for references involving profiling of
> various methods, including more modern techniques, with an application
> in general model-fitting.
>
>
>
>
> [1] http://tolstoy.newcastle.edu.au/R/help/00b/2492.html
>
> [2] Numerical Optimization, 2nd ed
> _________________________________________________________________
> 100?s of Music vouchers to be won with MSN Music
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>