R help - Apr 2020 - repeat rows of matrix by number (k) in one colummatrix adding column j with values 1:k

Hi

I can achieve this using two for loops but it is slow  I need to do this on
many matrices with tens of millions of rows of x,y,z and k

What is a faster method to achieve this, I cannot use rep as j changes in
each row of the new matrix
###############################################
M<-matrix(c(1,2,3,4,1,2,3,4,1,2,3,4, 2, 1, 3, 2
), 4,4, dimnames = list(NULL, c("x", "y",
"z","k")))

Print(M)
#Create matrix (Mout) in this case 8 rows with x,y,z in each row of M
#repeated k times with column j numbered from 1:k
# ! can do with nested loops but this is very slow ( example below)
#How do I acheive this quickly without loops?
Mout<-NULL

for(i in 1:nrow(M)){
  a=M[i,c("x","y","z")]
  for (j in 1:M[i,"k"]){
    b=c(a,j)
    Mout<-rbind(Mout,b)
    }
}
colnames(Mout)[4]<-"j"
print(Mout)

#########################################################

Thanks

Nevil Amos

	[[alternative HTML version deleted]]

False premise: rep works fine

Mout2 <- cbind(M[ rep(seq.int(nrow(M)),M[,"k"]),
c("x","y","z")],unlist(lapply(M[,"k"],seq.int)))

On March 31, 2020 6:18:37 PM PDT, nevil amos <nevil.amos at gmail.com>
wrote:
>Hi
>
>I can achieve this using two for loops but it is slow  I need to do
>this on
>many matrices with tens of millions of rows of x,y,z and k
>
>What is a faster method to achieve this, I cannot use rep as j changes
>in
>each row of the new matrix
>###############################################
>M<-matrix(c(1,2,3,4,1,2,3,4,1,2,3,4, 2, 1, 3, 2
>), 4,4, dimnames = list(NULL, c("x", "y",
"z","k")))
>
>Print(M)
>#Create matrix (Mout) in this case 8 rows with x,y,z in each row of M
>#repeated k times with column j numbered from 1:k
># ! can do with nested loops but this is very slow ( example below)
>#How do I acheive this quickly without loops?
>Mout<-NULL
>
>for(i in 1:nrow(M)){
>  a=M[i,c("x","y","z")]
>  for (j in 1:M[i,"k"]){
>    b=c(a,j)
>    Mout<-rbind(Mout,b)
>    }
>}
>colnames(Mout)[4]<-"j"
>print(Mout)
>
>#########################################################
>
>Thanks
>
>Nevil Amos
>
>	[[alternative HTML version deleted]]
>
>______________________________________________
>R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.
-- 
Sent from my phone. Please excuse my brevity.

Well,
I found a way to do it partly using rep(), and one loop that makes it 10x
or more faster however would still be good to do without the loop at all
matrix made slightly beigger (10000 rows):

M<-matrix(c(1:30000
), 10000,3)
M<-cbind(M,sample(1:5,size = 10000,replace = T))
#Print(M)
#Create matrix (Mout) in this case 8 rows with x,y,z in each row of M
#repeated k times with column j numbered from 1:k
# ! can do with nested loops but this is very slow ( example below)
#How do I acheive this quickly without loops?
print("double loop")
print(system.time({
Mout<-NULL

for(i in 1:nrow(M)){
  a=M[i,1:3]
  k=M[i,4]
  for (j in 1:k){
    b=c(a,j)
    Mout<-rbind(Mout,b)
    }
}
colnames(Mout)<-c("x","y","z","j")
}))

print("rep and single loop")
print(system.time({
  j<-NULL
  MOut<-NULL
  MOut<-M[,1:3][rep(1:nrow(M), times = M[,4]), ]
  for(i in M[,4])(j<-c(j,1:i))
  MOut<-cbind(MOut,j)
  colnames(Mout)<-c("x","y","z","j")
}))

On Wed, 1 Apr 2020 at 12:18, nevil amos <nevil.amos at gmail.com> wrote:
> Hi
>
> I can achieve this using two for loops but it is slow  I need to do this
> on many matrices with tens of millions of rows of x,y,z and k
>
> What is a faster method to achieve this, I cannot use rep as j changes in
> each row of the new matrix
> ###############################################
> M<-matrix(c(1,2,3,4,1,2,3,4,1,2,3,4, 2, 1, 3, 2
> ), 4,4, dimnames = list(NULL, c("x", "y",
"z","k")))
>
> Print(M)
> #Create matrix (Mout) in this case 8 rows with x,y,z in each row of M
> #repeated k times with column j numbered from 1:k
> # ! can do with nested loops but this is very slow ( example below)
> #How do I acheive this quickly without loops?
> Mout<-NULL
>
> for(i in 1:nrow(M)){
>   a=M[i,c("x","y","z")]
>   for (j in 1:M[i,"k"]){
>     b=c(a,j)
>     Mout<-rbind(Mout,b)
>     }
> }
> colnames(Mout)[4]<-"j"
> print(Mout)
>
> #########################################################
>
> Thanks
>
> Nevil Amos
>
	[[alternative HTML version deleted]]

OK sorted - hope these postings might help someone else

Any even faster options would be appreciated still

#seq() does not work but  sequence() does
print("rep and sequence")
print(system.time({
  j<-NULL
  MOut<-NULL
  MOut<-M[rep(1:nrow(M), times = M[,4]), ]
  j<-sequence(M[,4])
  MOut<-cbind(MOut,j)
  colnames(Mout)<-c("x","y","z","j")
}))

On Wed, 1 Apr 2020 at 13:42, nevil amos <nevil.amos at gmail.com> wrote:
> Well,
> I found a way to do it partly using rep(), and one loop that makes it 10x
> or more faster however would still be good to do without the loop at all
> matrix made slightly beigger (10000 rows):
>
> M<-matrix(c(1:30000
> ), 10000,3)
> M<-cbind(M,sample(1:5,size = 10000,replace = T))
> #Print(M)
> #Create matrix (Mout) in this case 8 rows with x,y,z in each row of M
> #repeated k times with column j numbered from 1:k
> # ! can do with nested loops but this is very slow ( example below)
> #How do I acheive this quickly without loops?
> print("double loop")
> print(system.time({
> Mout<-NULL
>
> for(i in 1:nrow(M)){
>   a=M[i,1:3]
>   k=M[i,4]
>   for (j in 1:k){
>     b=c(a,j)
>     Mout<-rbind(Mout,b)
>     }
> }
>
colnames(Mout)<-c("x","y","z","j")
> }))
>
> print("rep and single loop")
> print(system.time({
>   j<-NULL
>   MOut<-NULL
>   MOut<-M[,1:3][rep(1:nrow(M), times = M[,4]), ]
>   for(i in M[,4])(j<-c(j,1:i))
>   MOut<-cbind(MOut,j)
>  
colnames(Mout)<-c("x","y","z","j")
> }))
>
> On Wed, 1 Apr 2020 at 12:18, nevil amos <nevil.amos at gmail.com>
wrote:
>
>> Hi
>>
>> I can achieve this using two for loops but it is slow  I need to do
this
>> on many matrices with tens of millions of rows of x,y,z and k
>>
>> What is a faster method to achieve this, I cannot use rep as j changes
in
>> each row of the new matrix
>> ###############################################
>> M<-matrix(c(1,2,3,4,1,2,3,4,1,2,3,4, 2, 1, 3, 2
>> ), 4,4, dimnames = list(NULL, c("x", "y",
"z","k")))
>>
>> Print(M)
>> #Create matrix (Mout) in this case 8 rows with x,y,z in each row of M
>> #repeated k times with column j numbered from 1:k
>> # ! can do with nested loops but this is very slow ( example below)
>> #How do I acheive this quickly without loops?
>> Mout<-NULL
>>
>> for(i in 1:nrow(M)){
>>   a=M[i,c("x","y","z")]
>>   for (j in 1:M[i,"k"]){
>>     b=c(a,j)
>>     Mout<-rbind(Mout,b)
>>     }
>> }
>> colnames(Mout)[4]<-"j"
>> print(Mout)
>>
>> #########################################################
>>
>> Thanks
>>
>> Nevil Amos
>>
>
	[[alternative HTML version deleted]]

Hi Nevil,
It's a nasty piece of work, but:

M<-matrix(c(1,2,3,4,1,2,3,4,1,2,3,4,2,1,3,2),4,4,
dimnames = list(NULL, c("x", "y",
"z","k")))
M
reprow<-function(x)
 return(matrix(rep(x,x[length(x)]),nrow=x[length(x)],byrow=TRUE))
toseq<-function(x) return(1:x)
j<-unlist(sapply(M[,"k"],toseq))
Mlist<-apply(M,1,reprow)
Mlist
newM<-Mlist[[1]]
for(i in 2:length(Mlist)) newM<-rbind(newM,Mlist[[i]])
newM
newM<-cbind(newM,j)
colnames(newM)<-c(colnames(M),"j")
newM

Jim

On Wed, Apr 1, 2020 at 1:43 PM nevil amos <nevil.amos at gmail.com>
wrote:
>
> Well,
> I found a way to do it partly using rep(), and one loop that makes it 10x
> or more faster however would still be good to do without the loop at all
> matrix made slightly beigger (10000 rows):
>
> M<-matrix(c(1:30000
> ), 10000,3)
> M<-cbind(M,sample(1:5,size = 10000,replace = T))
> #Print(M)
> #Create matrix (Mout) in this case 8 rows with x,y,z in each row of M
> #repeated k times with column j numbered from 1:k
> # ! can do with nested loops but this is very slow ( example below)
> #How do I acheive this quickly without loops?
> print("double loop")
> print(system.time({
> Mout<-NULL
>
> for(i in 1:nrow(M)){
>   a=M[i,1:3]
>   k=M[i,4]
>   for (j in 1:k){
>     b=c(a,j)
>     Mout<-rbind(Mout,b)
>     }
> }
>
colnames(Mout)<-c("x","y","z","j")
> }))
>
> print("rep and single loop")
> print(system.time({
>   j<-NULL
>   MOut<-NULL
>   MOut<-M[,1:3][rep(1:nrow(M), times = M[,4]), ]
>   for(i in M[,4])(j<-c(j,1:i))
>   MOut<-cbind(MOut,j)
>  
colnames(Mout)<-c("x","y","z","j")
> }))
>
> On Wed, 1 Apr 2020 at 12:18, nevil amos <nevil.amos at gmail.com>
wrote:
>
> > Hi
> >
> > I can achieve this using two for loops but it is slow  I need to do
this
> > on many matrices with tens of millions of rows of x,y,z and k
> >
> > What is a faster method to achieve this, I cannot use rep as j changes
in
> > each row of the new matrix
> > ###############################################
> > M<-matrix(c(1,2,3,4,1,2,3,4,1,2,3,4, 2, 1, 3, 2
> > ), 4,4, dimnames = list(NULL, c("x", "y",
"z","k")))
> >
> > Print(M)
> > #Create matrix (Mout) in this case 8 rows with x,y,z in each row of M
> > #repeated k times with column j numbered from 1:k
> > # ! can do with nested loops but this is very slow ( example below)
> > #How do I acheive this quickly without loops?
> > Mout<-NULL
> >
> > for(i in 1:nrow(M)){
> >   a=M[i,c("x","y","z")]
> >   for (j in 1:M[i,"k"]){
> >     b=c(a,j)
> >     Mout<-rbind(Mout,b)
> >     }
> > }
> > colnames(Mout)[4]<-"j"
> > print(Mout)
> >
> > #########################################################
> >
> > Thanks
> >
> > Nevil Amos
> >
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.