R devel - Sep 2007 - Bug with Cor(..., method='spearman") and by() (PR#9921)

I posted this on R help, and a few others responded indicating they too
were able to replicate the error as a function of missing data. I
believe this should not be the case and hence and reporting it here.

### Code provided on R-Help by Ivar Herfindal
# Simulate data
testdata <- cbind.data.frame(gr=3Drep(letters[1:4], each=3D5),
aa=3Drnorm(20),
bb=3Drnorm(20))
# Introduce some missingness
testdata[1:5, 2] <- NA

# This works fine
by(testdata[,c("aa", "bb")], testdata$gr, cor,
use=3D"complete",
method=3D"pearson")

# This induces error
by(testdata[,c("aa", "bb")], testdata$gr, cor,
use=3D"complete",
method=3D"spearman")

Error in FUN(data[x, ], ...) : 'x' is empty

## Alternatively, we can try this

# This works fine
by(testdata[,c('aa', 'bb')], testdata$gr, cor,
use=3D'complete',
method=3D'pearson')

## This induces the same error
by(testdata[,c('aa', 'bb')], testdata$gr, cor,
use=3D'complete',
method=3D'spearman')

Error in FUN(data[x, ], ...) : 'x' is empty

I am using Windows XP with session info below

Harold Doran

> sessionInfo()
R version 2.5.0 (2007-04-23)=20
i386-pc-mingw32=20

locale:
LC_COLLATE=3DEnglish_United States.1252;LC_CTYPE=3DEnglish_United
States.1252;LC_MONETARY=3DEnglish_United
States.1252;LC_NUMERIC=3DC;LC_TIME=3DEnglish_United States.1252

attached base packages:
[1] "stats"     "graphics"  "grDevices"
"utils"     "datasets"
"methods"   "base"    =20

other attached packages:
     mlmRev        lme4      Matrix     lattice=20
  "0.995-1" "0.99875-2" "0.99875-3"   
"0.15-4"=20

This is nothing whatsoever to do with by(), and it is cor, not Cor.
Try

X <- cbind(NA, 1:3)
cor(X, use = "complete")
cor(X, use = "complete", method="spearman")

In short, cor() behaves differently when given a vector of NAs.  That's 
perfectly reasonable, as the ranks are undefined.

Since
> cor(na.omit(X))
Error in cor(na.omit(X)) : 'x' is empty

I would say that consistency requires that both examples give an error.

On Thu, 20 Sep 2007, HDoran at air.org wrote:
> I posted this on R help, and a few others responded indicating they too
> were able to replicate the error as a function of missing data. I
> believe this should not be the case and hence and reporting it here.
But you have failed to explain to us why such an example must work: your 
'belief' is not relevant here.  As the FAQ says, do not report as a bug 
something you do not 'know for certain'.
> ### Code provided on R-Help by Ivar Herfindal
> # Simulate data
> testdata <- cbind.data.frame(gr=3Drep(letters[1:4], each=3D5), >
aa=3Drnorm(20),
> bb=3Drnorm(20))
> # Introduce some missingness
> testdata[1:5, 2] <- NA
>
> # This works fine
> by(testdata[,c("aa", "bb")], testdata$gr, cor,
use=3D"complete",
> method=3D"pearson")
>
> # This induces error
> by(testdata[,c("aa", "bb")], testdata$gr, cor,
use=3D"complete",
> method=3D"spearman")
>
> Error in FUN(data[x, ], ...) : 'x' is empty
>
> ## Alternatively, we can try this
>
> # This works fine
> by(testdata[,c('aa', 'bb')], testdata$gr, cor,
use=3D'complete',
> method=3D'pearson')
>
> ## This induces the same error
> by(testdata[,c('aa', 'bb')], testdata$gr, cor,
use=3D'complete',
> method=3D'spearman')
>
> Error in FUN(data[x, ], ...) : 'x' is empty
>
> I am using Windows XP with session info below
>
> Harold Doran
>
>
>> sessionInfo()
> R version 2.5.0 (2007-04-23)=20
> i386-pc-mingw32=20
>
> locale:
> LC_COLLATE=3DEnglish_United States.1252;LC_CTYPE=3DEnglish_United
> States.1252;LC_MONETARY=3DEnglish_United
> States.1252;LC_NUMERIC=3DC;LC_TIME=3DEnglish_United States.1252
>
> attached base packages:
> [1] "stats"     "graphics"  "grDevices"
"utils"     "datasets"
> "methods"   "base"    =20
>
> other attached packages:
>     mlmRev        lme4      Matrix     lattice=20
>  "0.995-1" "0.99875-2" "0.99875-3"   
"0.15-4"=20
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595