R help - Dec 2002 - acceptable p-level for scientific studies

Dear list members,

I have a statistical question, that doesn't belong to this list, and I
apologise for that in advance but I would appreciate your help very much.
Is there some convention for selecting the a level for significance testing
in scientific (e.g. chemical processes) studies? Most people use the 0.05
level but I could not find a reference to justify this. Why not 0.01 or 0.1?
Montgomery in his book "Design and Analysis of Experiments" disagrees
with
setting a priori acceptable levels at all. Is it necessary to set a limit
for significance testing since R can provide exact probability levels for
the significance of each effect?

Thanks in advance.

Kyriakos Kachrimanis.

On Wed, 18 Dec 2002 14:44:05 +0200
Kyriakos Kachrimanis <kgk at pharm.auth.gr> wrote:
> Dear list members,
> 
> I have a statistical question, that doesn't belong to this list, and I
> apologise for that in advance but I would appreciate your help very much.
> Is there some convention for selecting the a level for significance testing
> in scientific (e.g. chemical processes) studies? Most people use the 0.05
> level but I could not find a reference to justify this. Why not 0.01 or
0.1?
> Montgomery in his book "Design and Analysis of Experiments"
disagrees with
> setting a priori acceptable levels at all. Is it necessary to set a limit
> for significance testing since R can provide exact probability levels for
> the significance of each effect?
> 
> Thanks in advance.
> 
> Kyriakos Kachrimanis.
> 
Want to open up the floodgates?  Some personal opinions:

- Alpha=0.05 is arbitrary, silly, and boring
- Reporting P and letting the reader decide is a bit better
- Bayesian posterior probabilities are still better although
  more thinking is required
- Confidence limits can be good compromise solutions (some journals are
  almost disallowing P-values in favor of CLs)
- P-values are dangerous, especially large, small, and in-between ones.
  See http://hesweb1.med.virginia.edu/biostat/teaching/bayes.short.course.pdf
for a full sermon.
-- 
Frank E Harrell Jr              Prof. of Biostatistics & Statistics
Div. of Biostatistics & Epidem. Dept. of Health Evaluation Sciences
U. Virginia School of Medicine  http://hesweb1.med.virginia.edu/biostat

For some further discussion, see
Nickerson, Raymond S. Null hypothesis significance testing: A
review of an old and continuing controversy. [Journal Article]
Psychological Methods. Vol 5(2) Jun 2000, 241-301.