R help - Jun 2005 - (Off topic.) Observed Fisher information.

I have been building an R function to calculate the ***observed***
(as opposed to expected) Fisher information matrix for parameter
estimates in a rather complicated setting.  I thought I had it
working, but I am getting a result which is not positive definite.
(One negative eigenvalue.  Out of 10.)

Is it the case that the observed Fisher information must be positive
definite --- thereby indicating for certain that there are errors in
my code --- or is it possible for such a matrix not to be pos. def.?

It seems to me that if the log likelihood surface is ***not*** well
approximated by a quadratic in a neighbourhood of the maximum, then
it might well be that case that the observed information could fail
to be positive definite.  Is this known/understood?  Can anyone point
me to appropriate places in the literature?

TIA.
			cheers,

				Rolf Turner

Hi Rolf,

If your data come from exponential family of distributions, then the
log-likelihood is concave and the observed information must be positive
definite.  However, I don't think that this is the case more generally, i.e.
for families such as curved exponential families the log-likelihood doesn't
have to concave.  I remember reading something about this in
Barndorff-Nielsen and Cox's book on Inference and Asymptotics.  There may be
better references.

Ravi.

--------------------------------------------------------------------------
Ravi Varadhan, Ph.D.
Assistant Professor,  The Center on Aging and Health
Division of Geriatric Medicine and Gerontology
Johns Hopkins University
Ph: (410) 502-2619
Fax: (410) 614-9625
Email:  rvaradhan at jhmi.edu
--------------------------------------------------------------------------
> -----Original Message-----
> From: r-help-bounces at stat.math.ethz.ch [mailto:r-help-
> bounces at stat.math.ethz.ch] On Behalf Of Rolf Turner
> Sent: Monday, June 06, 2005 6:49 PM
> To: r-help at stat.math.ethz.ch
> Subject: [R] (Off topic.) Observed Fisher information.
> 
> I have been building an R function to calculate the ***observed***
> (as opposed to expected) Fisher information matrix for parameter
> estimates in a rather complicated setting.  I thought I had it
> working, but I am getting a result which is not positive definite.
> (One negative eigenvalue.  Out of 10.)
> 
> Is it the case that the observed Fisher information must be positive
> definite --- thereby indicating for certain that there are errors in
> my code --- or is it possible for such a matrix not to be pos. def.?
> 
> It seems to me that if the log likelihood surface is ***not*** well
> approximated by a quadratic in a neighbourhood of the maximum, then
> it might well be that case that the observed information could fail
> to be positive definite.  Is this known/understood?  Can anyone point
> me to appropriate places in the literature?
> 
> TIA.
> 			cheers,
> 
> 				Rolf Turner
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! http://www.R-project.org/posting-
> guide.html

If you compute the observed information for sigma from a sample of size 1
from a N(0,sigma^2) distribution, you will find that the observed information
can be negative definite with a high probability. So it can happen!

Cheers!
John Holt


-----------------------------------------------------
I have been building an R function to calculate the ***observed***
(as opposed to expected) Fisher information matrix for parameter
estimates in a rather complicated setting.  I thought I had it
working, but I am getting a result which is not positive definite.
(One negative eigenvalue.  Out of 10.)

Is it the case that the observed Fisher information must be positive
definite --- thereby indicating for certain that there are errors in
my code --- or is it possible for such a matrix not to be pos. def.?

It seems to me that if the log likelihood surface is ***not*** well
approximated by a quadratic in a neighbourhood of the maximum, then
it might well be that case that the observed information could fail
to be positive definite.  Is this known/understood?  Can anyone point
me to appropriate places in the literature?

TIA.
cheers,

Rolf Turner
-----------------------------------------------------


John Holt, Ph.D.
Dept. Mathematics and Statistics
University of Guelph
Guelph, ON
N1G 2W1

Tel 519-824-4120Ext53297/52155

Rolf Turner wrote:
> I have been building an R function to calculate the ***observed***
> (as opposed to expected) Fisher information matrix for parameter
> estimates in a rather complicated setting.  I thought I had it
> working, but I am getting a result which is not positive definite.
> (One negative eigenvalue.  Out of 10.)
> 
> Is it the case that the observed Fisher information must be positive
> definite --- thereby indicating for certain that there are errors in
> my code --- or is it possible for such a matrix not to be pos. def.?
If you are at the maximum, it should be at least positive indefinite (or 
nonnegative definite).  Numerical errors could make zero (or small 
positive) eigenvalues look negative.  It's also possible that your 
optimization has missed the maximum by a bit, and then it could have 
truly negative eigenvalues.

In either case I'd expect the negative eigenvalues to be
small.
> 
> It seems to me that if the log likelihood surface is ***not*** well
> approximated by a quadratic in a neighbourhood of the maximum, then
> it might well be that case that the observed information could fail
> to be positive definite.  Is this known/understood?  Can anyone point
> me to appropriate places in the literature?
If there is a true negative eigenvalue, then moving along that 
eigenvector should increase the likelihood, so I don't think even 
irregular problems could have true negative eigenvalues at the MLE. The 
problem there would be that a zero score and a positive definite 
observed information matrix don't necessarily imply you're at even a 
local maximum.

Duncan Murdoch