R help - Jun 2006 - MLE maximum number of parameters

Hi All,

I would like to know, is there a *ballpark* figure for how many  
parameters the minimisation routines can cope with?

I'm asking because I was asked if I knew.

Cheers,

Federico

--
Federico C. F. Calboli
Department of Epidemiology and Public Health
Imperial College, St. Mary's Campus
Norfolk Place, London W2 1PG

Tel +44 (0)20 75941602   Fax +44 (0)20 75943193

f.calboli [.a.t] imperial.ac.uk
f.calboli [.a.t] gmail.com

Federico Calboli <f.calboli <at> imperial.ac.uk> writes:
> 
> Hi All,
> 
> I would like to know, is there a *ballpark* figure for how many  
> parameters the minimisation routines can cope with?
> 
  I think I would make a distinction between theoretical
and practical limits.  A lot depends on how fast your
objective function is, but I would say in general that
if your objective function has more than a few tens
of parameters you should probably be looking for some
more specialized optimization code (e.g. AD Model Builder,
finding a way to shoehorn it into existing routines
such as nlme/lmer, etc.).
   I would be happy to be corrected though.

  Ben

It really depends on how well-behaved your objective function is, but I've
been
able to fit a few models with 10--15 parameters.  But I felt like I was 
stretching the limit there.

-roger

Federico Calboli wrote:
> Hi All,
> 
> I would like to know, is there a *ballpark* figure for how many  
> parameters the minimisation routines can cope with?
> 
> I'm asking because I was asked if I knew.
> 
> Cheers,
> 
> Federico
> 
> --
> Federico C. F. Calboli
> Department of Epidemiology and Public Health
> Imperial College, St. Mary's Campus
> Norfolk Place, London W2 1PG
> 
> Tel +44 (0)20 75941602   Fax +44 (0)20 75943193
> 
> f.calboli [.a.t] imperial.ac.uk
> f.calboli [.a.t] gmail.com
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html
> 
-- 
Roger D. Peng  |  http://www.biostat.jhsph.edu/~rpeng/

Seagulls have a very different perspective to ballparks
than ants.  Nonetheless, there is something that can be
said.

There are several variables in addition to the number of
parameters that are important.  These include:

* The complexity of the likelihood

* The number of observations in the dataset

* How close to the optimum is close enough

* Your patience

The latter is undoubtedly the most important of all.  It
matters a lot whether you think a minute is a long time
or only periods measured in weeks.

The optimization strategy can also have a big effect.  If
you are using a derivative-based optimizer, then the number
of parameters can have a big impact.  Typically one iteration
in such algorithms requires p+1 function calls, where p is the
number of parameters.  Since more iterations are generally
required with more parameters, the speed can decrease
rapidly as the number of parameters increases.

One strategy to deal with a large number of parameters is to
start with something like a genetic algorithm.  Once the genetic
algorithm has a pretty good solution, then switch to a derivative-
based algorithm to finish.  The amount to run the initial
algorithm before switching depends on the problem, the quality
of the two optimizers, and probably other things.

With this switching strategy and at least a modicum of patience,
problems with thousands of parameters may be feasible to solve.

Patrick Burns
patrick at burns-stat.com
+44 (0)20 8525 0696
http://www.burns-stat.com
(home of S Poetry and "A Guide for the Unwilling S User")

Federico Calboli wrote:
>Hi All,
>
>I would like to know, is there a *ballpark* figure for how many  
>parameters the minimisation routines can cope with?
>
>I'm asking because I was asked if I knew.
>
>Cheers,
>
>Federico
>
>--
>Federico C. F. Calboli
>Department of Epidemiology and Public Health
>Imperial College, St. Mary's Campus
>Norfolk Place, London W2 1PG
>
>Tel +44 (0)20 75941602   Fax +44 (0)20 75943193
>
>f.calboli [.a.t] imperial.ac.uk
>f.calboli [.a.t] gmail.com
>
>______________________________________________
>R-help at stat.math.ethz.ch mailing list
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html
>
>
>  
>