R help - Dec 2011 - Improve a browse through list items - Transform a loop to apply-able function.

Un texte encapsul? et encod? dans un jeu de caract?res inconnu a ?t? nettoy?...
Nom : non disponible
URL :
<https://stat.ethz.ch/pipermail/r-help/attachments/20111212/5611c696/attachment.pl>

Robin Cura wrote on 12/12/2011 10:15:32 AM:
> Hello,
> 
> I'm currently trying to convert a slow and ugly script I made, so that 
it's
> faster and can be computed on a computer grid with the multicore 
package.
> My problem is that I don't see how to turn some loops into an 
"apply-able"
> function.
> 
> Here's an example of my loops :
> I got a list of dataframes (or matrices like here), and I need to browse
> each cell of those many dataframes to compute a mean (or standard 
deviation
> like here).
> 
> Here's a example script :
> 
> a <- b <- c <- d <- result <- matrix(nrow=3, ncol=3)
> a[] <- sample.int(n=100,size=9,replace=TRUE)
> b[] <- sample.int(n=100,size=9,replace=TRUE)
> c[] <- sample.int(n=100,size=9,replace=TRUE)
> d[] <- sample.int(n=100,size=9,replace=TRUE)
> result[] <- NA
> mylist <- list(a,b,c,d)
> 
> for (row in 1:3)
> {
>   for (col in 1:3)
>   {
>     tmpList <- log(mylist[[1]][row, col])
>     for (listitem in 2:4)
>     {
>       tmpList <- c(tmpList, log(mylist[[listitem]][row, col]))
>     }
>     result[row, col] <- sd(tmpList)
>   }
> }
> 
> Considering I have to look at the same cell in each dataframe, I don't
> understand how I could turn this into a function, considering I need the
> row and column number to iterate.
> 
> I succeeded improving my script duration a lot, but such loops are 
really
> long to run, considering that my lists contains like 100 dataframes, who
> all contains thousands of values.
> 
> Any help would be really appreciated
> 
> Thanks in advance,
> 
> Robin

Since your matrices are all the same dimension, your data would be easier 
to handle if they were in an array.

# convert list of matrices to a single array
L <- length(mylist)
RC <- dim(mylist[[1]])
myarray <- array(unlist(mylist), dim=c(RC[1], RC[2], L))

# then your script can be whittled down to a single line
apply(log(myarray), 1:2, sd)


Jean
	[[alternative HTML version deleted]]

> -----Original Message-----
> From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-
> project.org] On Behalf Of Robin Cura
> Sent: Tuesday December 13, 2011 3:16 AM
> 
> I'm currently trying to convert a slow and ugly script I made, so that
> it's
> faster and can be computed on a computer grid with the multicore
> package.
> My problem is that I don't see how to turn some loops into an
"apply-
> able"
> function.
> 
> 
> Here's a example script :
> 
> a <- b <- c <- d <- result <- matrix(nrow=3, ncol=3)
> a[] <- sample.int(n=100,size=9,replace=TRUE)
> b[] <- sample.int(n=100,size=9,replace=TRUE)
> c[] <- sample.int(n=100,size=9,replace=TRUE)
> d[] <- sample.int(n=100,size=9,replace=TRUE)
> result[] <- NA
> mylist <- list(a,b,c,d)
> 
> for (row in 1:3)
> {
>   for (col in 1:3)
>   {
>     tmpList <- log(mylist[[1]][row, col])
>     for (listitem in 2:4)
>     {
>       tmpList <- c(tmpList, log(mylist[[listitem]][row, col]))
>     }
>     result[row, col] <- sd(tmpList)
>   }
> }
> 
> Considering I have to look at the same cell in each dataframe, I don't
> understand how I could turn this into a function, considering I need
> the
> row and column number to iterate.
> 
How about something along the line of 

library(foreach)
library(doMC)
registerDoMC()   

# larger matrix for timing test
a <- b <- c <- d <- result <- matrix(nrow=1000, ncol=1000)
a[] <- sample.int(n=100,size=1000000,replace=TRUE)
b[] <- sample.int(n=100,size=1000000,replace=TRUE)
c[] <- sample.int(n=100,size=1000000,replace=TRUE)
d[] <- sample.int(n=100,size=1000000,replace=TRUE)
result[] <- NA
mylist <- list(a,b,c,d)
nrows=nrow(a)
ncols=ncol(a)

system.time(
{
  result<-foreach (row=1:nrows, .combine=rbind) %dopar%
  {
    thisrow=vector()

    for (col in 1:ncols)
    {
      tmpList <- log(mylist[[1]][row, col])
      for (listitem in 2:length(mylist))
      {
        tmpList <- c(tmpList, log(mylist[[listitem]][row, col]))
      }
      thisrow <- cbind(thisrow,sd(tmpList))
    }
    thisrow
  }
}
)

Example as written with the larger matrix
   user  system elapsed 
 62.660   0.000  62.722

%do%
   user  system elapsed 
 66.910   0.020  66.979 

%dopar%
   user  system elapsed 
 71.390   4.840   3.402


Klint.

Hi robin,
I'm not sure is what you need, but that's an esthetically nice
solution (one single line without any loop :) )

matrix(apply(log(cbind(as.numeric(a),as.numeric(b),as.numeric(c),as.numeric(d))),1,sd),3)

hope it could help,

PF


On Mon, Dec 12, 2011 at 5:15 PM, Robin Cura <robin.cura at gmail.com>
wrote:
> Hello,
>
> I'm currently trying to convert a slow and ugly script I made, so that
it's
> faster and can be computed on a computer grid with the multicore package.
> My problem is that I don't see how to turn some loops into an
"apply-able"
> function.
>
> Here's an example of my loops :
> I got a list of dataframes (or matrices like here), and I need to browse
> each cell of those many dataframes to compute a mean (or standard deviation
> like here).
>
> Here's a example script :
>
> a <- b <- c <- d <- result <- matrix(nrow=3, ncol=3)
> a[] <- sample.int(n=100,size=9,replace=TRUE)
> b[] <- sample.int(n=100,size=9,replace=TRUE)
> c[] <- sample.int(n=100,size=9,replace=TRUE)
> d[] <- sample.int(n=100,size=9,replace=TRUE)
> result[] <- NA
> mylist <- list(a,b,c,d)
>
> for (row in 1:3)
> {
> ?for (col in 1:3)
> ?{
> ? ?tmpList <- log(mylist[[1]][row, col])
> ? ?for (listitem in 2:4)
> ? ?{
> ? ? ?tmpList <- c(tmpList, log(mylist[[listitem]][row, col]))
> ? ?}
> ? ?result[row, col] <- sd(tmpList)
> ?}
> }
>
> Considering I have to look at the same cell in each dataframe, I don't
> understand how I could turn this into a function, considering I need the
> row and column number to iterate.
>
> I succeeded improving my script duration a lot, but such loops are really
> long to run, considering that my lists contains like 100 dataframes, who
> all contains thousands of values.
>
> Any help would be really appreciated
>
> Thanks in advance,
>
> Robin
>
> ? ? ? ?[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.


-- 
+-----------------------------------------------------------------------
| Patrizio Frederic,
| http://www.economia.unimore.it/frederic_patrizio/
+-----------------------------------------------------------------------

Un texte encapsul? et encod? dans un jeu de caract?res inconnu a ?t? nettoy?...
Nom : non disponible
URL :
<https://stat.ethz.ch/pipermail/r-help/attachments/20111216/28892b4f/attachment.pl>