R help - Mar 2012 - resampling for correlation and testing

Hello all R-er,

I'm trying to run a resampling method on some data. The current method I
have takes 2+ days or a lot of memory . I was wondering if anyone has a better
suggestion.

Currently I take a matrix and get the correlation matrix from it. This will be
called rho.A. Each element in this will be tested against the distribution from
the resampled correlation B matrix.

Some example code:

A<-matrix(rnorm(100), ncol=10)
B<-matrix(rnorm(100), ncol=10)

rho.A<-cor(A)

{
idx<-sample(1:10, 10)
idx
# [1]  8  4  5  7  1  9  2 10  6  3

rho.B<-cor(B[,idx])
} ## repeat this x time (currently 500)

## in essence we then have the following :
rho.arrayB<-array(runif((10*10)*500), dim=c(10,10,500))

## Then test if rho.A[1,1] come from the distribution of rho.B[1,1]
pvalueMat[1,1]<-wilcox.test(rho.array[1,1,] , rho.A[1,1])$p.value

However, my array size would be 2300 x 2300 x 500 which R won't let me even
make as an empty structure. Any suggestion are more than welcomed !!

Cheers,

Paul

On Wed, Mar 28, 2012 at 3:53 PM, Benton, Paul
<hpaul.benton08 at imperial.ac.uk> wrote:
> Hello all R-er,
>
> I'm trying to run a resampling method on some data. The current method
I have takes 2+ days or a lot of memory . I was wondering if anyone has a better
suggestion.
>
> Currently I take a matrix and get the correlation matrix from it. This will
be called rho.A. Each element in this will be tested against the distribution
from the resampled correlation B matrix.
>
> Some example code:
>
> A<-matrix(rnorm(100), ncol=10)
> B<-matrix(rnorm(100), ncol=10)
>
> rho.A<-cor(A)
>
> {
> idx<-sample(1:10, 10)
> idx
> # [1] 8 4 5 7 1 9 2 10 6 ?3
>
> rho.B<-cor(B[,idx])
> } ## repeat this x time (currently 500)
>
> ## in essence we then have the following :
> rho.arrayB<-array(runif((10*10)*500), dim=c(10,10,500))
Err... no we don't. sample(10,10) ; sample(10,10) ... only permutes
the columns, so the 500 cor(B) have exactly the same values in
different off diag positions. Using runif they are
unique
>
> ## Then test if rho.A[1,1] come from the distribution of rho.B[1,1]
> pvalueMat[1,1]<-wilcox.test(rho.array[1,1,] , rho.A[1,1])$p.value
>
>From what I know cor(A)[ i , i ] = cor(B)[ j , j ] = 1   for any
choice of A,B,i and j
I don't think Wilcox intended his test to be used in this way....

I would start with fixing these issues first so you don't wait 2 days
for a vector of NaN's

Cheers

> However, my array size would be 2300 x 2300 x 500 which R won't let me
even make as an empty structure. Any suggestion are more than welcomed !!
>
> Cheers,
>
> Paul
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.