Dear Anne In addition to Marc's comments if you are forced to do this then, assuming your package computes sample size from power then just feed it a range of powers and find the one for which it calculates the sample size you had. There is a more elegant way to do this using uniroot but brute force should work. Michael On 26/08/2019 13:42, Marc Schwartz via R-help wrote:> >> On Aug 26, 2019, at 6:24 AM, CHATTON Anne via R-help <r-help at r-project.org> wrote: >> >> Hello everybody, >> >> I am trying to accommodate the R codes provided by Donohue for sample size calculation in the package "longpower" with lmmpower function to estimate the post-hoc power (asked by a reviewer) of a binary GEE model with a three-way interaction (time x condition x continuous predictor) given a fixed sample size. In other words instead of the sample size I would like to estimate the power of my study. >> >> Could anyone please help me to modify these codes as to obtain the power I'm looking for. >> >> I would really appreciate receiving any feedback on this subject. >> >> Yours sincerely, >> >> Anne > > > Hi, > > Three comments: > > 1. Don't calculate post hoc power. Do a Google search and you will find a plethora of papers and discussions on why not, including these: > > The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis > The American Statistician, February 2001, Vol. 55, No. 1 > https://www.vims.edu/people/hoenig_jm/pubs/hoenig2.pdf > > Post Hoc Power: Tables and Commentary > https://stat.uiowa.edu/sites/stat.uiowa.edu/files/techrep/tr378.pdf > > Observed power, and what to do if your editor asks for post-hoc power analyses > http://daniellakens.blogspot.com/2014/12/observed-power-and-what-to-do-if-your.html > > Retraction Watch: > Statisticians clamor for retraction of paper by Harvard researchers they say uses a ?nonsense statistic? > https://retractionwatch.com/2019/06/19/statisticians-clamor-for-retraction-of-paper-by-harvard-researchers-they-say-uses-a-nonsense-statistic/ > > PubPeer Comments on the paper cited in the above RW post: > https://pubpeer.com/publications/4399282A80691D9421B497E8316CF6 > > A discussion on Frank's Data Methods forum also related to the same paper cited above: > "Observed Power" and other "Power" Issues > https://discourse.datamethods.org/t/observed-power-and-other-power-issues/731/30 > > > 2. If you are still compelled (voluntarily or involuntarily), you may want to review the vignette for the longpower package which may have some insights, and/or contact the package maintainer for additional guidance on how to structure the code. See the vignette here: > > https://cran.r-project.org/web/packages/longpower/vignettes/longpower.pdf > > > 3. Don't calculate post hoc power. > > > Regards, > > Marc Schwartz > > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > > --- > This email has been checked for viruses by AVG. > https://www.avg.com > >-- Michael http://www.dewey.myzen.co.uk/home.html
That doesn't work. In caricature, post-hoc power is - I observe a difference of nearly zero - However, to find a significant difference of that size I'd need 200000 observations - I only used 100 observations - Therefore my study is useless and can be discarded (or: I calculate the probability of Type II error if the true difference is the observed and get 0.999... Therefore, etc.) Best way out is a confidence interval. Second best (but in principle wrong) is to redo the pre-study power calculation and say that the study was designed to find a difference of delta, which it clearly didn't, so the true difference is probably less than delta. -pd> On 26 Aug 2019, at 18:29 , Michael Dewey <lists at dewey.myzen.co.uk> wrote: > > Dear Anne > > In addition to Marc's comments if you are forced to do this then, assuming your package computes sample size from power then just feed it a range of powers and find the one for which it calculates the sample size you had. There is a more elegant way to do this using uniroot but brute force should work. > > Michael > > On 26/08/2019 13:42, Marc Schwartz via R-help wrote: >>> On Aug 26, 2019, at 6:24 AM, CHATTON Anne via R-help <r-help at r-project.org> wrote: >>> >>> Hello everybody, >>> >>> I am trying to accommodate the R codes provided by Donohue for sample size calculation in the package "longpower" with lmmpower function to estimate the post-hoc power (asked by a reviewer) of a binary GEE model with a three-way interaction (time x condition x continuous predictor) given a fixed sample size. In other words instead of the sample size I would like to estimate the power of my study. >>> >>> Could anyone please help me to modify these codes as to obtain the power I'm looking for. >>> >>> I would really appreciate receiving any feedback on this subject. >>> >>> Yours sincerely, >>> >>> Anne >> Hi, >> Three comments: >> 1. Don't calculate post hoc power. Do a Google search and you will find a plethora of papers and discussions on why not, including these: >> The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis >> The American Statistician, February 2001, Vol. 55, No. 1 >> https://www.vims.edu/people/hoenig_jm/pubs/hoenig2.pdf >> Post Hoc Power: Tables and Commentary >> https://stat.uiowa.edu/sites/stat.uiowa.edu/files/techrep/tr378.pdf >> Observed power, and what to do if your editor asks for post-hoc power analyses >> http://daniellakens.blogspot.com/2014/12/observed-power-and-what-to-do-if-your.html >> Retraction Watch: >> Statisticians clamor for retraction of paper by Harvard researchers they say uses a ?nonsense statistic? >> https://retractionwatch.com/2019/06/19/statisticians-clamor-for-retraction-of-paper-by-harvard-researchers-they-say-uses-a-nonsense-statistic/ >> PubPeer Comments on the paper cited in the above RW post: >> https://pubpeer.com/publications/4399282A80691D9421B497E8316CF6 >> A discussion on Frank's Data Methods forum also related to the same paper cited above: >> "Observed Power" and other "Power" Issues >> https://discourse.datamethods.org/t/observed-power-and-other-power-issues/731/30 >> 2. If you are still compelled (voluntarily or involuntarily), you may want to review the vignette for the longpower package which may have some insights, and/or contact the package maintainer for additional guidance on how to structure the code. See the vignette here: >> https://cran.r-project.org/web/packages/longpower/vignettes/longpower.pdf >> 3. Don't calculate post hoc power. >> Regards, >> Marc Schwartz >> ______________________________________________ >> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> --- >> This email has been checked for viruses by AVG. >> https://www.avg.com > > -- > Michael > http://www.dewey.myzen.co.uk/home.html > > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.-- Peter Dalgaard, Professor, Center for Statistics, Copenhagen Business School Solbjerg Plads 3, 2000 Frederiksberg, Denmark Phone: (+45)38153501 Office: A 4.23 Email: pd.mes at cbs.dk Priv: PDalgd at gmail.com
Dear Michael, Thanks a lot for your suggestion. This is what I am trying to do with R (longpower and gee packages). But I am getting stuck with a confusing error message sent earlier I don't understand. Best, Anne -----Message d'origine----- De?: Michael Dewey [mailto:lists at dewey.myzen.co.uk] Envoy??: lundi, 26 ao?t 2019 18:29 ??: Marc Schwartz <marc_schwartz at me.com>; CHATTON Anne <Anne.Chatton at hcuge.ch> Cc?: R-help <r-help at r-project.org> Objet?: Re: [R] Code modification for post-hoc power Dear Anne In addition to Marc's comments if you are forced to do this then, assuming your package computes sample size from power then just feed it a range of powers and find the one for which it calculates the sample size you had. There is a more elegant way to do this using uniroot but brute force should work. Michael On 26/08/2019 13:42, Marc Schwartz via R-help wrote:> >> On Aug 26, 2019, at 6:24 AM, CHATTON Anne via R-help <r-help at r-project.org> wrote: >> >> Hello everybody, >> >> I am trying to accommodate the R codes provided by Donohue for sample size calculation in the package "longpower" with lmmpower function to estimate the post-hoc power (asked by a reviewer) of a binary GEE model with a three-way interaction (time x condition x continuous predictor) given a fixed sample size. In other words instead of the sample size I would like to estimate the power of my study. >> >> Could anyone please help me to modify these codes as to obtain the power I'm looking for. >> >> I would really appreciate receiving any feedback on this subject. >> >> Yours sincerely, >> >> Anne > > > Hi, > > Three comments: > > 1. Don't calculate post hoc power. Do a Google search and you will find a plethora of papers and discussions on why not, including these: > > The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis > The American Statistician, February 2001, Vol. 55, No. 1 > https://www.vims.edu/people/hoenig_jm/pubs/hoenig2.pdf > > Post Hoc Power: Tables and Commentary > https://stat.uiowa.edu/sites/stat.uiowa.edu/files/techrep/tr378.pdf > > Observed power, and what to do if your editor asks for post-hoc power analyses > > http://daniellakens.blogspot.com/2014/12/observed-power-and-what-to-do > -if-your.html > > Retraction Watch: > Statisticians clamor for retraction of paper by Harvard researchers they say uses a ?nonsense statistic? > > https://retractionwatch.com/2019/06/19/statisticians-clamor-for-retrac > tion-of-paper-by-harvard-researchers-they-say-uses-a-nonsense-statisti > c/ > > PubPeer Comments on the paper cited in the above RW post: > https://pubpeer.com/publications/4399282A80691D9421B497E8316CF6 > > A discussion on Frank's Data Methods forum also related to the same paper cited above: > "Observed Power" and other "Power" Issues > > https://discourse.datamethods.org/t/observed-power-and-other-power-iss > ues/731/30 > > > 2. If you are still compelled (voluntarily or involuntarily), you may want to review the vignette for the longpower package which may have some insights, and/or contact the package maintainer for additional guidance on how to structure the code. See the vignette here: > > > https://cran.r-project.org/web/packages/longpower/vignettes/longpower. > pdf > > > 3. Don't calculate post hoc power. > > > Regards, > > Marc Schwartz > > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > > --- > This email has been checked for viruses by AVG. > https://www.avg.com > >-- Michael http://www.dewey.myzen.co.uk/home.html
Dear Anne Can you resend the eror message which you accidentally sent only to me please? Michael On 27/08/2019 08:02, CHATTON Anne wrote:> Dear Michael, > > Thanks a lot for your suggestion. This is what I am trying to do with R (longpower and gee packages). But I am getting stuck with a confusing error message sent earlier I don't understand. > > Best, > > Anne > > -----Message d'origine----- > De?: Michael Dewey [mailto:lists at dewey.myzen.co.uk] > Envoy??: lundi, 26 ao?t 2019 18:29 > ??: Marc Schwartz <marc_schwartz at me.com>; CHATTON Anne <Anne.Chatton at hcuge.ch> > Cc?: R-help <r-help at r-project.org> > Objet?: Re: [R] Code modification for post-hoc power > > Dear Anne > > In addition to Marc's comments if you are forced to do this then, assuming your package computes sample size from power then just feed it a range of powers and find the one for which it calculates the sample size you had. There is a more elegant way to do this using uniroot but brute force should work. > > Michael > > On 26/08/2019 13:42, Marc Schwartz via R-help wrote: >> >>> On Aug 26, 2019, at 6:24 AM, CHATTON Anne via R-help <r-help at r-project.org> wrote: >>> >>> Hello everybody, >>> >>> I am trying to accommodate the R codes provided by Donohue for sample size calculation in the package "longpower" with lmmpower function to estimate the post-hoc power (asked by a reviewer) of a binary GEE model with a three-way interaction (time x condition x continuous predictor) given a fixed sample size. In other words instead of the sample size I would like to estimate the power of my study. >>> >>> Could anyone please help me to modify these codes as to obtain the power I'm looking for. >>> >>> I would really appreciate receiving any feedback on this subject. >>> >>> Yours sincerely, >>> >>> Anne >> >> >> Hi, >> >> Three comments: >> >> 1. Don't calculate post hoc power. Do a Google search and you will find a plethora of papers and discussions on why not, including these: >> >> The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis >> The American Statistician, February 2001, Vol. 55, No. 1 >> https://www.vims.edu/people/hoenig_jm/pubs/hoenig2.pdf >> >> Post Hoc Power: Tables and Commentary >> https://stat.uiowa.edu/sites/stat.uiowa.edu/files/techrep/tr378.pdf >> >> Observed power, and what to do if your editor asks for post-hoc power analyses >> >> http://daniellakens.blogspot.com/2014/12/observed-power-and-what-to-do >> -if-your.html >> >> Retraction Watch: >> Statisticians clamor for retraction of paper by Harvard researchers they say uses a ?nonsense statistic? >> >> https://retractionwatch.com/2019/06/19/statisticians-clamor-for-retrac >> tion-of-paper-by-harvard-researchers-they-say-uses-a-nonsense-statisti >> c/ >> >> PubPeer Comments on the paper cited in the above RW post: >> https://pubpeer.com/publications/4399282A80691D9421B497E8316CF6 >> >> A discussion on Frank's Data Methods forum also related to the same paper cited above: >> "Observed Power" and other "Power" Issues >> >> https://discourse.datamethods.org/t/observed-power-and-other-power-iss >> ues/731/30 >> >> >> 2. If you are still compelled (voluntarily or involuntarily), you may want to review the vignette for the longpower package which may have some insights, and/or contact the package maintainer for additional guidance on how to structure the code. See the vignette here: >> >> >> https://cran.r-project.org/web/packages/longpower/vignettes/longpower. >> pdf >> >> >> 3. Don't calculate post hoc power. >> >> >> Regards, >> >> Marc Schwartz >> >> ______________________________________________ >> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide >> http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> >> --- >> This email has been checked for viruses by AVG. >> https://www.avg.com >> >> > > -- > Michael > http://www.dewey.myzen.co.uk/home.html >-- Michael http://www.dewey.myzen.co.uk/home.html