R help - Sep 2004 - Multiple comparisons in a non parametric case

Dear all,

I am conducting a full factorial analysis. I have one factor consisting
in algorithms, which I consider my treatments, and another factor made
of the problems I want to solve. For each problem I obtain a response
variable which is stochastic. I replicate the measure of this response
value 10 times.

When I apply ANOVA the assumptions do not hold, hence I must rely on non
parametric tests.

By transforming the response data in ranks, the Friedman test tells me
that there is statistical significance in the difference of the sum of
ranks of at least one of the treatments.

I would like now to produce a plot for the multiple comparisons similar
to the Least Significant Difference or the Tukey's Honest Significant
Difference used in ANOVA. Since I am in the non parametric case I can
not use these methods.

Instead, I compare graphically individual treatments by plotting the sum
of ranks of each treatment togehter with the 95% confidence interval. To
compute the interval I use the Friedman test as suggested by Conover in
"Practical Nonparametric statistics".

I obtain something like this:

Treat. A                |-+-|
Treat. B              |-+-|
Treat. C                   |-+-|
Treat. D           |-+-|

The intervals have all the same spread because the number of
replications was the same for all experimental units.


I would like to know if someone in the list had a similar experience and
if what I am doing is correct. In alternative also a reference to
another list which could better fit my request is welcome.


Thank you for the help,

Marco



--
Marco Chiarandini, Fachgebiet Intellektik, Fachbereich Informatik,
Technische Universit??t Darmstadt, Hochschulstra??e 10,
D-64289 Darmstadt - Germany, Office: S2/02 Raum E317
Tel: +49 (0)6151 16-6802 Fax: +49 (0)6151 16-5326
email: machud at intellektik.informatik.tu-darmstadt.de
web page: http://www.intellektik.informatik.tu-darmstadt.de/~machud

It looks to me like what you are doing is trying to judge
significance of differences by non-overlap of single-sample
confidence intervals.  While this is appealing, it's not quite
right.

I just looked into my copy of Applied Nonparametric Statistics
(second ed.) by Wayne W. Daniel (Duxbury, 1990) but that
only deals with the situation where there is a single replicate
per block-treatment combination (whereas you have 10 reps)
and block-treatment interaction is assumed to be non-existent.

The method that Daniel prescribes in this simple setting seems to be
no more than applying the Bonferroni method of multiple comparisons.
(Daniel does not say; his book is very much a cook-book.)  So you
might simply try Bonferroni --- i.e. do all k-choose-2 pairwise
comparisons between treatments (using the appropriate 2 sample method
for each comparison) doing each comparison at the alpha/k-choose-2
significance level.  Where k = the number of treatments = 4 in your
case.  This method is not going to be super-powerful but it is
sometimes surprizing how well Bonferroni stacks up against more
``sophisticated'' methods.

Daniel gives a reference to ``Nonparametric Statistical Methods'' by
Myles Hollander and Douglas A. Wolfe, New York, Wiley, 1973, for ``an
alternative multiple comparisons formula''.  I don't have this book,
and don't know what direction Hollander and Wolfe ride off in, but it
***might*** be worth trying to get your hands on it and see.

Finally --- in what way are the assumptions of Anova violated?  The
conventional wisdom is that Anova is actually quite robust to
non-normality.  Particularly when the sample size is large --- and 10
reps per treatment combination is pretty good.  Heteroskedasticity is
more of a worry, but it's not so much of a worry when the design is
nicely balanced.  As yours is.  And finally-finally --- have you
tried transforming your data to make them a bit more normal and/or
homoskedastic?

I hope this is some help.

				cheers,

					Rolf Turner
					rolf at math.unb.ca

Marco Chiarandini wrote:
> I am conducting a full factorial analysis. I have one factor
> consisting in algorithms, which I consider my treatments, and another
> factor made of the problems I want to solve. For each problem I
> obtain a response variable which is stochastic. I replicate the
> measure of this response value 10 times.
> 
> When I apply ANOVA the assumptions do not hold, hence I must rely on
> non parametric tests.
> 
> By transforming the response data in ranks, the Friedman test tells
> me that there is statistical significance in the difference of the
> sum of ranks of at least one of the treatments.
> 
> I would like now to produce a plot for the multiple comparisons
> similar to the Least Significant Difference or the Tukey's Honest
> Significant Difference used in ANOVA. Since I am in the non
> parametric case I can not use these methods.
> 
> Instead, I compare graphically individual treatments by plotting the
> sum of ranks of each treatment togehter with the 95% confidence
> interval. To compute the interval I use the Friedman test as
> suggested by Conover in "Practical Nonparametric statistics".
> 
> I obtain something like this:
> 
> Treat. A                |-+-|
> Treat. B              |-+-|
> Treat. C                   |-+-|
> Treat. D           |-+-|
> 
> The intervals have all the same spread because the number of
> replications was the same for all experimental units.
> 
> I would like to know if someone in the list had a similar experience
> and if what I am doing is correct. In alternative also a reference to
> another list which could better fit my request is welcome.