R help - Dec 2007 - Reproducibility of experiment

Dear list,

I have an experiment that I have run 10 times in order to find out its 
reproducibility. I wonder if there is any function that I can use for 
obtaining a significance value of reproducibility or agreement of 
measurements. I thought of coefficient of variation but, as far as I 
know, I would have to set a threshold for saying the experiment is not 
reproducible. Any pointers to something more "objective" would be very
helpful.

Thanks

David

On Wed, 2007-12-12 at 17:23 +0100, darteta001 at ikasle.ehu.es
wrote:
> Dear list,
> 
> I have an experiment that I have run 10 times in order to find out its 
> reproducibility. I wonder if there is any function that I can use for 
> obtaining a significance value of reproducibility or agreement of 
> measurements. I thought of coefficient of variation but, as far as I 
> know, I would have to set a threshold for saying the experiment is not 
> reproducible. Any pointers to something more "objective" would be
very
> helpful.
> 
> Thanks
> 
> David
I suspect that you are going to have to be more specific regarding the
subject matter and the experimental design so that those with the
requisite expertise could comment.

If this is looking at a continuous measure (ie. instrumentation
measurements), you could look at Bland-Altman methods. More information
here:

  http://www-users.york.ac.uk/~mb55/meas/meas.htm

Otherwise, given that Google returns almost a million hits with the
phrase "reproducibility of experiment"...

HTH,

Marc Schwartz

Dear Marc and R-list,

thanks for your help. I 
have checked Bland-Altman help 
page about repeatability, and I learnt that instead of 
reproducibility, 
I was talking about repeatability. Although I am not sure whether they 
only 
focuse on agreement of two different measurement methods, and not 
on 
repeatability of one single method. 

To explain further on my topic, I have repeated ten times an 
experiment involving protein quantification(i.e. how much protein I 
have), 
giving me ten continuous values. All experimental settings are similar 
so there should be no variability due to day of experiment, operator
or any batch effect. My aim is to know whether these ten observations 
are good enough so that I can conclude that the repeatability of my 
detection technique is good. But as I have learnt from Altman?s page, 
it is not possible to set a threshold to the repeatability score to 
say my experiment is "repeatable".  
I guess I can obtain a 95% confidence interval for the protein 
quantification values, 
but I am not sure this will show how well my experiment performs. 
Putting it differently, 
something I would like to know is whether I can estimate
beforehand how many times I need to run an experiment in order to be 
confident that it is "repeatable".


Thanks for your comments

David
> 
> On Wed, 2007-12-12 at 17:23 +0100, darteta001 at ikasle.ehu.es wrote:
> > Dear list,
> > 
> > I have an experiment that I have run 10 times in order to find out 
its 
> > reproducibility. I wonder if there is any function that I can use 
for 
> > obtaining a significance value of reproducibility or agreement of 
> > measurements. I thought of coefficient of variation but, as far as 
I 
> > know, I would have to set a threshold for saying the experiment is 
not 
> > reproducible. Any pointers to something more "objective"
would be
very 
> > helpful.
> > 
> > Thanks
> > 
> > David
> 
> I suspect that you are going to have to be more specific regarding 
the
> subject matter and the experimental design so that those with the
> requisite expertise could comment.
> 
> If this is looking at a continuous measure (ie. instrumentation
> measurements), you could look at Bland-Altman methods. More 
information
> here:
> 
>   http://www-users.york.ac.uk/~mb55/meas/meas.htm
> 
> Otherwise, given that Google returns almost a million hits with the
> phrase "reproducibility of experiment"...
> 
> HTH,
> 
> Marc Schwartz
> 
> 
>