My apologies, I forgot to CC: to the list on my previous communication
with Daniel.
Jesse
----- Forwarded by Jesse Albert Canchola/EMVL/DIAG/US/BAYER on 08/21/2006
10:50 AM -----
Jesse Albert Canchola/EMVL/DIAG/US/BAYER
08/21/2006 09:36 AM
To
"Daniel Nordlund" <res90sx5 at verizon.net>
cc
Subject
RE: [R] Permutations with replacement
Thanks, Daniel. I need to enumerate all possibilities of 8^8 and take a
random sample of 10,000 from there. Then I will use the sampled
possibilities to do a combination of data frames/files then do some math
on those files and develop the probability distribution from resulting
sampling statistics (I couldn't get the available bootstrap packages to do
what I want). BTW, the preferred solution (however inelegant) is
reprinted below. I did have a memory problem with the hypercube for 8^8
so I did an 8^7 hypercube that worked and concatenated a 1-8 to the 8^7
matrix that resulted in 8 large matrices which I attempted rbind together
to create the 8^8 but ran into more memory problems (on matrix number 7 of
8- it's a Windows problem - I used the --max-mem-size=2G - to no avail).
The final solution was to take a random sample of 10,000/8=1250 from each
of the files (doing four of the 8 files at a time, and for which I
permuted the rows to make it more random), removed the heavy-laden files
then rbind 'ed the smaller sampled files together to make the 10,000.
Here is the final final code:
# IDEA: We cannot simply do a permutation of the 8 classes/file id's
since this will not allow/simulate repeats of numbers
# as in a bootstrap (e.g., for 3 items - 1,2,3 - we would also want
the possibility 1,1,1 or 2,2,2 or 3,3,3 etc.
# The 8!=40,320 permutations with replacement would become an
8^8=16,777,216 so we would want to take a random
# sample of 10,000 from the posibilities
library(combinat)
# THIS WILL NOT WORK DUE TO THE LIMITATIONS OF WINDOWS MEMORY (PROBLEMATIC
AS IN THE FAQs) SO WE WILL
# USE A WORKAROUND
# WORKAROUND: 1) Construct the 8 to the 7 power hypercube. 2) For these
data, create eight additional data sets that include
# the last position to finish the construction of an 8^8
hypercube
#x <- rep(8,8) # for partitions of 8 units into classes {1,2,3,4,5,6,7,8}
#hcube8 <- hcube(x, scale=1, transl=0)
#hcube8
#step 1: 8^7 = 2,097,152
x1 <- rep(8,7)
x1
hcube87 <- hcube(x1, scale=1, transl=0)
#this will generate 2,097,152 results from 1-8 but for only 7 positions
#step 2: column bind each file with 1-8 in the 8th position
#x1a <- cbind(hcube87,1)
#x2a <- cbind(hcube87,2)
#x3a <- cbind(hcube87,3)
#x4a <- cbind(hcube87,4)
#x5a <- cbind(hcube87,5)
#x6a <- cbind(hcube87,6)
#x7a <- cbind(hcube87,7)
#x8a <- cbind(hcube87,8)
#turns out this method also chokes with the memory limitations
# Step 2 will be modified as follows:
# Step 2a: as before; Step2b: Sample 1/8 from each piece and after every 4
processes, delete the objects to allow for the rest.
#Step 2a
x1a <- cbind(hcube87,1)
x1a <- x1a[sample(1:2097152),] #randomly permute the rows for more
randomness
x2a <- cbind(hcube87,2)
x2a <- x2a[sample(1:2097152),] #randomly permute the rows for more
randomness
x3a <- cbind(hcube87,3)
x3a <- x3a[sample(1:2097152),] #randomly permute the rows for more
randomness
x4a <- cbind(hcube87,4)
x4a <- x4a[sample(1:2097152),] #randomly permute the rows for more
randomness
x1b <- x1a[sample(1:1250,replace=FALSE),]
x2b <- x2a[sample(1:1250,replace=FALSE),]
x3b <- x3a[sample(1:1250,replace=FALSE),]
x4b <- x4a[sample(1:1250,replace=FALSE),]
rm(x1a,x2a,x3a,x4a) #remove the big files
x5a <- cbind(hcube87,5)
x5a <- x5a[sample(1:2097152),] #randomly permute the rows for more
randomness
x6a <- cbind(hcube87,6)
x6a <- x6a[sample(1:2097152),] #randomly permute the rows for more
randomness
x7a <- cbind(hcube87,7)
x7a <- x7a[sample(1:2097152),] #randomly permute the rows for more
randomness
x8a <- cbind(hcube87,8)
x8a <- x8a[sample(1:2097152),] #randomly permute the rows for more
randomness
x5b <- x5a[sample(1:1250,replace=FALSE),]
x6b <- x6a[sample(1:1250,replace=FALSE),]
x7b <- x7a[sample(1:1250,replace=FALSE),]
x8b <- x8a[sample(1:1250,replace=FALSE),]
rm(x5a,x6a,x7a,x8a) #remove the big files
#Step 3: combine all the randomly sampled files
m <- rbind(x1b,x2b,x3b,x4b,x5b,x6b,x7b,x8b)
# NOTE: each number in the matrix represents a file "name" from 1-8.
# the first pointer should be numeric and then subsequent as character
# since the first time you assign a number to a character in a matrix
# the rest of the numbers in the matrix are coerced to character
m[m==1]='a'; m[m=='2']='b';
m[m=='3']='c' ; m[m=='4']='d';
m[m=='5']='e' ;
m[m=='6']='f'; m[m=='7']='g' ;
m[m=='8']='h'
m
########### end R code ############
Thanks, David. That worked fabulously!
Here is the R code for the hypercube test example:
########## begin R code ############
library(combinat)
x <- rep(3,3) # for partitions of 3 units into the three classes {1,2,3}
hcube(x, scale=1, transl=0)
########### end R code ############
For the larger one I want (i.e., 8^8), I will take a random sample of
10,000 from the 16,777,216 possibilities.
Regards,
Jesse Canchola
<davidr at rhotrading.com>
Sent by: r-help-bounces at stat.math.ethz.ch
08/18/2006 01:33 PM
To
"Jesse Albert Canchola" <jesse.canchola.b at bayer.com>,
"r-help"
<r-help at stat.math.ethz.ch>
cc
Subject
Re: [R] Permutations with replacement
If you also want 1,1,1 and so on, the number of these is n^n,
(n choices for each of n slots.)
In that case, you could use hcube from combinat.
David L. Reiner
Rho Trading Securities, LLC
Chicago IL 60605
"Daniel Nordlund" <res90sx5 at verizon.net>
08/18/2006 05:16 PM
To
"'Jesse Albert Canchola'" <jesse.canchola.b at
bayer.com>, "'r-help'"
<r-help at stat.math.ethz.ch>
cc
Subject
RE: [R] Permutations with replacement
> -----Original Message-----
> From: r-help-bounces at stat.math.ethz.ch
[mailto:r-help-bounces at stat.math.ethz.ch]> On Behalf Of Jesse Albert Canchola
> Sent: Friday, August 18, 2006 1:02 PM
> To: r-help
> Subject: [R] Permutations with replacement
>
> Is there a simple function or process that will create permutations with
> replacement?
>
> I know that using the combinat package
>
> ###### begin R code ######
> > library(combinat)
> > m <- t(array(unlist(permn(3)), dim = c(3, 6)))
>
> # we can get the permutations, for example 3!=6
> # gives us
>
> > m
> [,1] [,2] [,3]
> [1,] 1 2 3
> [2,] 1 3 2
> [3,] 3 1 2
> [4,] 3 2 1
> [5,] 2 3 1
> [6,] 2 1 3
> ###### end R code ##########
>
> I'd like to include the "with replacement possibilities" such
as
>
> 1,1,3
> 1,1,2
> 2,3,3
>
Isn't what you want just sampling with replacement?
x <- c(1,2,3)
sample(x,3,replace=TRUE)
Hope this is helpful,
Dan
Dan Nordlund
Bothell, WA USA
_______________________________________________________________________________________________
The information contained in this e-mail is for the exclusive use of the
intended recipient(s) and may be confidential, proprietary, and/or legally
privileged. Inadvertent disclosure of this message does not constitute a waiver
of any privilege. If you receive this message in error, please do not directly
or indirectly use, print, copy, forward, or disclose any part of this message.
Please also delete this e-mail and all copies and notify the sender. Thank you.
For alternate languages please go to http://bayerdisclaimer.bayerweb.com
