R help - Sep 2005 - Off-topic: Comparing standard errors from simulation and analytical model

Dear list:

I'm hoping to tap in to the statistical expertise in the group,
especially those familiar with simulation techniques. I'm finalizing a
study where I obtain standard errors from two sources. The first source
is a monte carlo simulation and the other source is an analytical model
I have developed that appears to recover the standard errors from the
simulation. All analysis are performed in R using MASS, nlme, and
Matrix.

Here is a very brief description. In the monte carlo, I first sample
from a multivariate distribution to create data. The data are
hypothetical student scores on an achievement test over time and the aim
is to examine what happens to standard errors under certain psychometric
conditions. The data are then "contaminated" to reflect a certain
psychometric problem that occurs in longitudinal analyses of student
achievement scores.

These data are then analyzed using a linear model to obtain parameter
estimates. This is replicated 250 times. 

For example, the model equation used is

Y_{ti} =  \mu + \beta \cdot t + \epsilon_{ti}

So, I obtain 250 estimates of \mu and \beta. I take the standard
deviation of these estimates to get the sampling distribution of the
parameter (standard errors). Next, I take a single data set, contaminate
the scores, and then use the analytical approach to obtain standard
errors. So, I end up with two sets of standards errors, those obtained
under simulated conditions and those obtained from the analytical model.

My question is what are the most acceptable techniques for comparing the
standard errors in order to say that the analytical approach actually
"recovers" the monte carlo standard errors? For the most part, the
standard errors appear to be exactly the same, save rounding error.

One idea I am toying with is to average the standard errors of \mu and
\beta from the simulation and then do a t-test between the two standard
errors which might be something along these lines

t = (SE_{analytical} - SE_{mc} )/  \bar se

Where \bar se is the average of the standard errors.

But I'm not certain this is correct. Can anyone suggest a more
appropriate method for comparing the results?

Many thanks. I can also send a copy of the paper to anyone who would
like more information or details. 

-Harold

	[[alternative HTML version deleted]]

since you are interested especially in the standard errors, I think 
that you probably need something like a double simulation procedure, 
e.g.,

1. simulate data D[b] and "contaminate" them.

2. fit the model (with parameters \theta) using D[b], get \theta[b] 
and also compute the standard errors se.a[b] using the asymptotic 
method.

3. using \theta[b] simulate M new data sets, "contaminate" them, fit 
the model in each one, obtain \theta[m] and calculate the standard 
deviation of these estimates se.mc[b]

4. keep res[b] = (se.mc[b] - se.a[b]) / se.mc[b]

5. repeat steps 1-4 B times and calculate, e.g., a 95% CI for res 
using the sample quantiles.


of course this is going to be much more time consuming (depending on 
the choices of B and M), but I think it will give you better a picture 
of how your method performs.


I hope this helps.

Best,
Dimitris

----
Dimitris Rizopoulos
Ph.D. Student
Biostatistical Centre
School of Public Health
Catholic University of Leuven

Address: Kapucijnenvoer 35, Leuven, Belgium
Tel: +32/16/336899
Fax: +32/16/337015
Web: http://www.med.kuleuven.be/biostat/
     http://www.student.kuleuven.be/~m0390867/dimitris.htm


----- Original Message ----- 
From: "Doran, Harold" <HDoran at air.org>
To: <r-help at stat.math.ethz.ch>
Sent: Friday, September 09, 2005 4:03 PM
Subject: [R] Off-topic: Comparing standard errors from simulation 
andanalytical model

> Dear list:
>
> I'm hoping to tap in to the statistical expertise in the group,
> especially those familiar with simulation techniques. I'm finalizing 
> a
> study where I obtain standard errors from two sources. The first 
> source
> is a monte carlo simulation and the other source is an analytical 
> model
> I have developed that appears to recover the standard errors from 
> the
> simulation. All analysis are performed in R using MASS, nlme, and
> Matrix.
>
> Here is a very brief description. In the monte carlo, I first sample
> from a multivariate distribution to create data. The data are
> hypothetical student scores on an achievement test over time and the 
> aim
> is to examine what happens to standard errors under certain 
> psychometric
> conditions. The data are then "contaminated" to reflect a certain
> psychometric problem that occurs in longitudinal analyses of student
> achievement scores.
>
> These data are then analyzed using a linear model to obtain 
> parameter
> estimates. This is replicated 250 times.
>
> For example, the model equation used is
>
> Y_{ti} =  \mu + \beta \cdot t + \epsilon_{ti}
>
> So, I obtain 250 estimates of \mu and \beta. I take the standard
> deviation of these estimates to get the sampling distribution of the
> parameter (standard errors). Next, I take a single data set, 
> contaminate
> the scores, and then use the analytical approach to obtain standard
> errors. So, I end up with two sets of standards errors, those 
> obtained
> under simulated conditions and those obtained from the analytical 
> model.
>
> My question is what are the most acceptable techniques for comparing 
> the
> standard errors in order to say that the analytical approach 
> actually
> "recovers" the monte carlo standard errors? For the most part,
the
> standard errors appear to be exactly the same, save rounding error.
>
> One idea I am toying with is to average the standard errors of \mu 
> and
> \beta from the simulation and then do a t-test between the two 
> standard
> errors which might be something along these lines
>
> t = (SE_{analytical} - SE_{mc} )/  \bar se
>
> Where \bar se is the average of the standard errors.
>
> But I'm not certain this is correct. Can anyone suggest a more
> appropriate method for comparing the results?
>
> Many thanks. I can also send a copy of the paper to anyone who would
> like more information or details.
>
> -Harold
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! 
> http://www.R-project.org/posting-guide.html
> 

Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm