Bert Gunter
2018-Aug-22 14:55 UTC
[R] differing behavior of mean(), median() and sd() with na.rm
... And FWIW (not much, I agree), note that if z = numeric(0) and sum(z) 0, then mean(z) = NaN makes sense, as length(z) = 0, so dividing by 0 gives NaN. So you can see the sorts of issues you may need to consider. Bert Gunter "The trouble with having an open mind is that people keep coming along and sticking things into it." -- Opus (aka Berkeley Breathed in his "Bloom County" comic strip ) On Wed, Aug 22, 2018 at 7:47 AM Bert Gunter <bgunter.4567 at gmail.com> wrote:> Actually, the dissonance is a bit more basic. > > After xxx(...., na.rm=TRUE) with all NA's in ... you have numeric(0). So > what you see is actually: > > > z <- numeric(0) > > mean(z) > [1] NaN > > median(z) > [1] NA > > sd(z) > [1] NA > > sum(z) > [1] 0 > etc. > > I imagine that there may be more of these little inconsistencies due to > the organic way R evolved over time. What the conventions should be can be > purely a matter of personal opinion in the absence of accepted standards. > But I would look to see what accepted standards were, if any, first. > > -- Bert > > > On Wed, Aug 22, 2018 at 7:34 AM Ivan Calandra <calandra at rgzm.de> wrote: > >> Dear useRs, >> >> I have just noticed that when input is only NA with na.rm=TRUE, mean() >> results in NaN, whereas median() and sd() produce NA. Shouldn't it all >> be the same? I think NA makes more sense than NaN in that case. >> >> x <- c(NA, NA, NA) mean(x, na.rm=TRUE) [1] NaN median(x, na.rm=TRUE) [1] >> NAsd(x, na.rm=TRUE) [1] NA >> >> Thanks for any feedback. >> >> Best, >> Ivan >> >> -- >> Dr. Ivan Calandra >> TraCEr, laboratory for Traceology and Controlled Experiments >> MONREPOS Archaeological Research Centre and >> Museum for Human Behavioural Evolution >> Schloss Monrepos >> 56567 Neuwied, Germany >> +49 (0) 2631 9772-243 >> https://www.researchgate.net/profile/Ivan_Calandra >> >> ______________________________________________ >> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide >> http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> >[[alternative HTML version deleted]]
Marc Schwartz
2018-Aug-22 15:24 UTC
[R] differing behavior of mean(), median() and sd() with na.rm
Hi, It might even be worthwhile to review this recent thread on R-Devel: https://stat.ethz.ch/pipermail/r-devel/2018-July/076377.html which touches upon a subtly related topic vis-a-vis NaN handling. Regards, Marc Schwartz> On Aug 22, 2018, at 10:55 AM, Bert Gunter <bgunter.4567 at gmail.com> wrote: > > ... And FWIW (not much, I agree), note that if z = numeric(0) and sum(z) > 0, then mean(z) = NaN makes sense, as length(z) = 0, so dividing by 0 gives > NaN. So you can see the sorts of issues you may need to consider. > > Bert Gunter > > "The trouble with having an open mind is that people keep coming along and > sticking things into it." > -- Opus (aka Berkeley Breathed in his "Bloom County" comic strip ) > > > On Wed, Aug 22, 2018 at 7:47 AM Bert Gunter <bgunter.4567 at gmail.com> wrote: > >> Actually, the dissonance is a bit more basic. >> >> After xxx(...., na.rm=TRUE) with all NA's in ... you have numeric(0). So >> what you see is actually: >> >>> z <- numeric(0) >>> mean(z) >> [1] NaN >>> median(z) >> [1] NA >>> sd(z) >> [1] NA >>> sum(z) >> [1] 0 >> etc. >> >> I imagine that there may be more of these little inconsistencies due to >> the organic way R evolved over time. What the conventions should be can be >> purely a matter of personal opinion in the absence of accepted standards. >> But I would look to see what accepted standards were, if any, first. >> >> -- Bert >> >> >> On Wed, Aug 22, 2018 at 7:34 AM Ivan Calandra <calandra at rgzm.de> wrote: >> >>> Dear useRs, >>> >>> I have just noticed that when input is only NA with na.rm=TRUE, mean() >>> results in NaN, whereas median() and sd() produce NA. Shouldn't it all >>> be the same? I think NA makes more sense than NaN in that case. >>> >>> x <- c(NA, NA, NA) mean(x, na.rm=TRUE) [1] NaN median(x, na.rm=TRUE) [1] >>> NAsd(x, na.rm=TRUE) [1] NA >>> >>> Thanks for any feedback. >>> >>> Best, >>> Ivan >>> >>> -- >>> Dr. Ivan Calandra >>> TraCEr, laboratory for Traceology and Controlled Experiments >>> MONREPOS Archaeological Research Centre and >>> Museum for Human Behavioural Evolution >>> Schloss Monrepos >>> 56567 Neuwied, Germany >>> +49 (0) 2631 9772-243 >>> https://www.researchgate.net/profile/Ivan_Calandra >>> >>> ______________________________________________ >>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >>> https://stat.ethz.ch/mailman/listinfo/r-help >>> PLEASE do read the posting guide >>> http://www.R-project.org/posting-guide.html >>> and provide commented, minimal, self-contained, reproducible code. >>> >> > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.
Ted Harding
2018-Aug-22 16:41 UTC
[R] differing behavior of mean(), median() and sd() with na.rm
I think that one can usefully look at this question from the point of view of what "NaN" and "NA" are abbreviations for (at any rate, according to the understanding I have adopted since many years -- maybe over-simplified). NaN: Mot a Number NA: Not Available So NA is typically used for missing values, whereas NaN represents the reults of numerical calculations which cannot give a result which is a definite number, Hence 0/0 is not a number, so NaN; similarly Inf/Inf. Thus, with your x <- c(NA, NA, NA) mean(x, na.rm=TRUE) sum(x, na.rm=TRUE) = 0, since the set of values of x with na.rm=TRUE is empty so the number of elements in x is 0; hence mean = 0/0 = NaN. But for median(x, na.rm=TRUE), because there are no available elements in x with na.rm=TRUE, and the median is found by searching among available elements for the value which divides the set of values into two halves, the median is not available, hence NA. Best wishes to all, Ted. On Wed, 2018-08-22 at 11:24 -0400, Marc Schwartz via R-help wrote:> Hi, > > It might even be worthwhile to review this recent thread on R-Devel: > > https://stat.ethz.ch/pipermail/r-devel/2018-July/076377.html > > which touches upon a subtly related topic vis-a-vis NaN handling. > > Regards, > > Marc Schwartz > > > > On Aug 22, 2018, at 10:55 AM, Bert Gunter <bgunter.4567 at gmail.com> wrote: > > > > ... And FWIW (not much, I agree), note that if z = numeric(0) and sum(z) > > 0, then mean(z) = NaN makes sense, as length(z) = 0, so dividing by 0 gives > > NaN. So you can see the sorts of issues you may need to consider. > > > > Bert Gunter > > > > "The trouble with having an open mind is that people keep coming along and > > sticking things into it." > > -- Opus (aka Berkeley Breathed in his "Bloom County" comic strip ) > > > > > > On Wed, Aug 22, 2018 at 7:47 AM Bert Gunter <bgunter.4567 at gmail.com> wrote: > > > >> Actually, the dissonance is a bit more basic. > >> > >> After xxx(...., na.rm=TRUE) with all NA's in ... you have numeric(0). So > >> what you see is actually: > >> > >>> z <- numeric(0) > >>> mean(z) > >> [1] NaN > >>> median(z) > >> [1] NA > >>> sd(z) > >> [1] NA > >>> sum(z) > >> [1] 0 > >> etc. > >> > >> I imagine that there may be more of these little inconsistencies due to > >> the organic way R evolved over time. What the conventions should be can be > >> purely a matter of personal opinion in the absence of accepted standards. > >> But I would look to see what accepted standards were, if any, first. > >> > >> -- Bert > >> > >> > >> On Wed, Aug 22, 2018 at 7:34 AM Ivan Calandra <calandra at rgzm.de> wrote: > >> > >>> Dear useRs, > >>> > >>> I have just noticed that when input is only NA with na.rm=TRUE, mean() > >>> results in NaN, whereas median() and sd() produce NA. Shouldn't it all > >>> be the same? I think NA makes more sense than NaN in that case. > >>> > >>> x <- c(NA, NA, NA) mean(x, na.rm=TRUE) [1] NaN median(x, na.rm=TRUE) [1] > >>> NAsd(x, na.rm=TRUE) [1] NA > >>> > >>> Thanks for any feedback. > >>> > >>> Best, > >>> Ivan > >>> > >>> -- > >>> Dr. Ivan Calandra > >>> TraCEr, laboratory for Traceology and Controlled Experiments > >>> MONREPOS Archaeological Research Centre and > >>> Museum for Human Behavioural Evolution > >>> Schloss Monrepos > >>> 56567 Neuwied, Germany > >>> +49 (0) 2631 9772-243 > >>> https://www.researchgate.net/profile/Ivan_Calandra > >>> > >>> ______________________________________________ > >>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > >>> https://stat.ethz.ch/mailman/listinfo/r-help > >>> PLEASE do read the posting guide > >>> http://www.R-project.org/posting-guide.html > >>> and provide commented, minimal, self-contained, reproducible code. > >>> > >> > > > > [[alternative HTML version deleted]] > > > > ______________________________________________ > > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > > https://stat.ethz.ch/mailman/listinfo/r-help > > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > > and provide commented, minimal, self-contained, reproducible code. > > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.