R help - Aug 2009 - Likelihood Function for Multinomial Logistic Regression and its partial derivatives

Hi,

I would like to apply the L-BFGS optimization algorithm to compute the MLE
of a multilevel multinomial Logistic Regression. 

The likelihood formula for this model has as one of the summands the formula
for computing the likelihood of an ordinary (single-level) multinomial logit
regression. So I would basically need the R implementation for this formula.
The L-BFGS algorithm also requires computing the partial derivatives of that
formula in respect to all parameters. I would appreciate if you can point me
to existing implementations that can do the above.

Nick

PS. The long story for the above:

My data is as follows: 

- a vector of observed values (lenght = D) of the dependent multinomial
variable each element belonging to one of N levels of that variable

- a matrix of corresponding observed values (O x P) of the independent
variables (P in total, most of them are binary but also a few are
integer-valued)

- a vector of current estimates (or starting values) for the Beta
coefficients of the independent variables (length = P).

This data is available for 4 different pools. The partially-pooled model
that I want to compute has as a likelihood function a sum of several
elements, one being the classical likelihood function of a multinomial logit
regression for each of the 4 pools.

This is the same model as in Finkel and Manning "Hierarchical Bayesian
Domain Adaptation" (2009).

-- 
View this message in context:
http://www.nabble.com/Likelihood-Function-for-Multinomial-Logistic-Regression-and-its-partial-derivatives-tp24772731p24772731.html
Sent from the R help mailing list archive at Nabble.com.

Hi,

Providing the gradient function is generally a good idea in optimization;
however, it is not necessary.  Almost all optimization routines will compute
this using a simple finite-difference approximation, if they are not
user-specified. If your function is very complicated, then you are more likely
to make a mistake in computing analytic gradient, although many optimization
routines also provide a check to see if the gradient is correctly specified or
not.  But you can do this yourself using the `grad' function in
"numDeriv" package.

Hope this is helpful,
Ravi.

____________________________________________________________________

Ravi Varadhan, Ph.D.
Assistant Professor,
Division of Geriatric Medicine and Gerontology
School of Medicine
Johns Hopkins University

Ph. (410) 502-2619
email: rvaradhan at jhmi.edu


----- Original Message -----
From: nikolay12 <nikolay12 at gmail.com>
Date: Sunday, August 2, 2009 3:04 am
Subject: [R] Likelihood Function for Multinomial Logistic Regression and its
partial derivatives
To: r-help at r-project.org

>  Hi,
>  
>  I would like to apply the L-BFGS optimization algorithm to compute 
> the MLE
>  of a multilevel multinomial Logistic Regression. 
>  
>  The likelihood formula for this model has as one of the summands the 
> formula
>  for computing the likelihood of an ordinary (single-level) 
> multinomial logit
>  regression. So I would basically need the R implementation for this
formula.
>  The L-BFGS algorithm also requires computing the partial derivatives 
> of that
>  formula in respect to all parameters. I would appreciate if you can 
> point me
>  to existing implementations that can do the above.
>  
>  Nick
>  
>  PS. The long story for the above:
>  
>  My data is as follows: 
>  
>  - a vector of observed values (lenght = D) of the dependent multinomial
>  variable each element belonging to one of N levels of that variable
>  
>  - a matrix of corresponding observed values (O x P) of the independent
>  variables (P in total, most of them are binary but also a few are
>  integer-valued)
>  
>  - a vector of current estimates (or starting values) for the Beta
>  coefficients of the independent variables (length = P).
>  
>  This data is available for 4 different pools. The partially-pooled model
>  that I want to compute has as a likelihood function a sum of several
>  elements, one being the classical likelihood function of a 
> multinomial logit
>  regression for each of the 4 pools.
>  
>  This is the same model as in Finkel and Manning "Hierarchical
Bayesian
>  Domain Adaptation" (2009).
>  
>  -- 
>  View this message in context: 
>  Sent from the R help mailing list archive at Nabble.com.
>  
>  ______________________________________________
>  R-help at r-project.org mailing list
>  
>  PLEASE do read the posting guide 
>  and provide commented, minimal, self-contained, reproducible code.

Hi: John Fox's CAR book has some very nice examples of how the multinomial
   likelihood  is  estimated  computationally speaking. But you mentioned
   multilevel earlier which sounds more complex ?

   On Aug 2, 2009, nikolay12 <nikolay12 at gmail.com> wrote:

     Thanks a lot. The info about computing the gradient will be helpful.
     I admit that I am somewhat confused about the likelihood function itself.
     It
     is often said that you need to set a reference category. However, I found
     two different implementations in Matlab for which setting the reference
     category  is  not  really  obvious. Thus I wanted to find a single,
     indisputable
     implementation of the likelihood function for multinomous logistic
     regression in R. Can't believe there is no such available.
     Nick
     Ravi Varadhan wrote:
     >
     > Hi,
     >
     >  Providing  the  gradient  function  is  generally a good idea in
     optimization;
     > however, it is not necessary. Almost all optimization routines will
     > compute this using a simple finite-difference approximation, if they
are
     > not user-specified. If your function is very complicated, then you are
     > more likely to make a mistake in computing analytic gradient, although
     > many optimization routines also provide a check to see if the gradient
     is
     > correctly specified or not. But you can do this yourself using the
     `grad'
     > function in "numDeriv" package.
     >
     > Hope this is helpful,
     > Ravi.
     >
     > ____________________________________________________________________
     >
     > Ravi Varadhan, Ph.D.
     > Assistant Professor,
     > Division of Geriatric Medicine and Gerontology
     > School of Medicine
     > Johns Hopkins University
     >
     > Ph. (410) 502-2619
     > email: [1]rvaradhan at jhmi.edu
     >
     >
     > ----- Original Message -----
     > From: nikolay12 <[2]nikolay12 at gmail.com>
     > Date: Sunday, August 2, 2009 3:04 am
     > Subject: [R] Likelihood Function for Multinomial Logistic Regression
and
     > its partial derivatives
     > To: [3]r-help at r-project.org
     >
     >
     >> Hi,
     >>
     >> I would like to apply the L-BFGS optimization algorithm to compute
     >> the MLE
     >> of a multilevel multinomial Logistic Regression.
     >>
     >> The likelihood formula for this model has as one of the summands
the
     >> formula
     >> for computing the likelihood of an ordinary (single-level)
     >> multinomial logit
     >> regression. So I would basically need the R implementation for
this
     >> formula.
     >> The L-BFGS algorithm also requires computing the partial
derivatives
     >> of that
     >> formula in respect to all parameters. I would appreciate if you
can
     >> point me
     >> to existing implementations that can do the above.
     >>
     >> Nick
     >>
     >> PS. The long story for the above:
     >>
     >> My data is as follows:
     >>
     >> - a vector of observed values (lenght = D) of the dependent
multinomial
     >> variable each element belonging to one of N levels of that
variable
     >>
     >> - a matrix of corresponding observed values (O x P) of the
independent
     >> variables (P in total, most of them are binary but also a few are
     >> integer-valued)
     >>
     >> - a vector of current estimates (or starting values) for the Beta
     >> coefficients of the independent variables (length = P).
     >>
     >> This data is available for 4 different pools. The partially-pooled
     model
     >> that I want to compute has as a likelihood function a sum of
several
     >> elements, one being the classical likelihood function of a
     >> multinomial logit
     >> regression for each of the 4 pools.
     >>
     >> This is the same model as in Finkel and Manning "Hierarchical
Bayesian
     >> Domain Adaptation" (2009).
     >>
     >> --
     >> View this message in context:
     >> Sent from the R help mailing list archive at Nabble.com.
     >>
     >> ______________________________________________
     >> [4]R-help at r-project.org mailing list
     >>
     >> PLEASE do read the posting guide
     >> and provide commented, minimal, self-contained, reproducible code.
     >
     > ______________________________________________
     > [5]R-help at r-project.org mailing list
     > [6]https://stat.ethz.ch/mailman/listinfo/r-help
     > PLEASE do read the posting guide
     > [7]http://www.R-project.org/posting-guide.html
     > and provide commented, minimal, self-contained, reproducible code.
     >
     >
     --
     View this message in context:
     [8]http://www.nabble.com/Likelihood-Function-for-Multinomial-Logistic-Regr
     ession-and-its-partial-derivatives-tp24772731p24781298.html
     Sent from the R help mailing list archive at Nabble.com.
     ______________________________________________
     [9]R-help at r-project.org mailing list
     [10]https://stat.ethz.ch/mailman/listinfo/r-help
     PLEASE do read the posting guide
     [11]http://www.R-project.org/posting-guide.html
     and provide commented, minimal, self-contained, reproducible code.

References

   1. mailto:rvaradhan at jhmi.edu
   2. mailto:nikolay12 at gmail.com
   3. mailto:r-help at r-project.org
   4. mailto:R-help at r-project.org
   5. mailto:R-help at r-project.org
   6. https://stat.ethz.ch/mailman/listinfo/r-help
   7. http://www.R-project.org/posting-guide.html
   8.
http://www.nabble.com/Likelihood-Function-for-Multinomial-Logistic-Regression-and-its-partial-derivatives-tp24772731p24781298.html
   9. mailto:R-help at r-project.org
  10. https://stat.ethz.ch/mailman/listinfo/r-help
  11. http://www.R-project.org/posting-guide.html

You may refer to mlogit for the ordinary multinomial regression. As
fas as I know, there are no functions for multilevel multinomial
regression.

Ronggui

2009/8/2 nikolay12 <nikolay12 at gmail.com>:
>
> Hi,
>
> I would like to apply the L-BFGS optimization algorithm to compute the MLE
> of a multilevel multinomial Logistic Regression.
>
> The likelihood formula for this model has as one of the summands the
formula
> for computing the likelihood of an ordinary (single-level) multinomial
logit
> regression. So I would basically need the R implementation for this
formula.
> The L-BFGS algorithm also requires computing the partial derivatives of
that
> formula in respect to all parameters. I would appreciate if you can point
me
> to existing implementations that can do the above.
>
> Nick
>
> PS. The long story for the above:
>
> My data is as follows:
>
> - a vector of observed values (lenght = D) of the dependent multinomial
> variable each element belonging to one of N levels of that variable
>
> - a matrix of corresponding observed values (O x P) of the independent
> variables (P in total, most of them are binary but also a few are
> integer-valued)
>
> - a vector of current estimates (or starting values) for the Beta
> coefficients of the independent variables (length = P).
>
> This data is available for 4 different pools. The partially-pooled model
> that I want to compute has as a likelihood function a sum of several
> elements, one being the classical likelihood function of a multinomial
logit
> regression for each of the 4 pools.
>
> This is the same model as in Finkel and Manning "Hierarchical Bayesian
> Domain Adaptation" (2009).
>
> --
> View this message in context:
http://www.nabble.com/Likelihood-Function-for-Multinomial-Logistic-Regression-and-its-partial-derivatives-tp24772731p24772731.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>


-- 
HUANG Ronggui, Wincent
PhD Candidate
Dept of Public and Social Administration
City University of Hong Kong
Home page: http://asrr.r-forge.r-project.org/rghuang.html

Thanks, I had a look at mlogit. It seems it does fit a multinomial logit
regression but - just as nnet or VGAM are doing it - it has a function that
tells you the fitted value, not the value that you have with a set of
parameters (which might not be the optimal ones). Or am I wrong on this?


Ronggui Huang wrote:
> 
> You may refer to mlogit for the ordinary multinomial regression. As
> fas as I know, there are no functions for multilevel multinomial
> regression.
> 
> Ronggui
> 
-- 
View this message in context:
http://www.nabble.com/Likelihood-Function-for-Multinomial-Logistic-Regression-and-its-partial-derivatives-tp24772731p24784053.html
Sent from the R help mailing list archive at Nabble.com.