R help - Nov 2009 - Splitting massive output into multiple text files

Dear List,

I thought it would be much easier to put a second query into a second mail.

I need to print 426*10000 blocks of variance components data, where 426 is 
the number of columns of data that have 10000 permutations of variance 
generated for each of them.

I have tried printing out a smaller number of permutations for a smaller 
number of markers and that has worked.

However, since a text file will not handle 4 million blocks in a single 
file (which is what I ultimately need to do), is there a way to tell R to 
create a new file for every 10 or so columns?

I tried to use some suggested code that looked like:

for (j in 1:426)
{
 write(cbind(modeldf[,j:(j+9)]),file=as.character(j))
 j <- j+10
}

..but can't figure out how to put it into my own code and make it work.

I did find one example of code for split files, each successive file being 
labeled as a series of numbers, but I couldn't figure out how to even adapt 
that to my model.

I cannot figure out what other way there is to conveniently view 4 m. items 
of data without losing some of it somewhere..

Any help will be much much appreciated..

Aditi

------------------------------------------------------------------

Code for shorter sample file:


model<-read.table("...)

modeldf<-data.frame(model)

modeldf[2:13<-lapply(modeldf[2:13],factor)

colms<-(modeldf)[4:13]   ## 10 markers only in this file
se<-c(1:1000)

for(f in colms)
{
print("Marker")
{
for( i in 1:1000)
{
print("perm no.")
print(se[i])
{
peg.no.prm<-sample(peg.no, length(peg.no))
try(fit5<-lmer(data=modeldf, peg.no.prm~1 + (1|family/f)))
print(summary(fit5))
capture.output(fit5, file="testperm5.txt", append=T)
}}}
}


The data files are at:
>>
>>
>> <http://www.4shared.com/file/131980362/460bdafe/Testvcomp10.ht
>> ml> (excel)
>> http://www.4shared.com/file/131980512/dc7308b/Testvcomp10.html
>> (txt)
------------------------------------------------------------------


----------------------
A Singh
Aditi.Singh at bristol.ac.uk
School of Biological Sciences
University of Bristol

try:
capture.output(fit5, file=paste("testperm", i, ".txt",
sep=""), append=T)


Bart


A Singh wrote:
> 
> Dear List,
> 
> I thought it would be much easier to put a second query into a second
> mail.
> 
> I need to print 426*10000 blocks of variance components data, where 426 is 
> the number of columns of data that have 10000 permutations of variance 
> generated for each of them.
> 
> I have tried printing out a smaller number of permutations for a smaller 
> number of markers and that has worked.
> 
> However, since a text file will not handle 4 million blocks in a single 
> file (which is what I ultimately need to do), is there a way to tell R to 
> create a new file for every 10 or so columns?
> 
> I tried to use some suggested code that looked like:
> 
> for (j in 1:426)
> {
>  write(cbind(modeldf[,j:(j+9)]),file=as.character(j))
>  j <- j+10
> }
> 
> ..but can't figure out how to put it into my own code and make it work.
> 
> I did find one example of code for split files, each successive file being 
> labeled as a series of numbers, but I couldn't figure out how to even
> adapt 
> that to my model.
> 
> I cannot figure out what other way there is to conveniently view 4 m.
> items 
> of data without losing some of it somewhere..
> 
> Any help will be much much appreciated..
> 
> Aditi
> 
> ------------------------------------------------------------------
> 
> Code for shorter sample file:
> 
> 
> model<-read.table("...)
> 
> modeldf<-data.frame(model)
> 
> modeldf[2:13<-lapply(modeldf[2:13],factor)
> 
> colms<-(modeldf)[4:13]   ## 10 markers only in this file
> se<-c(1:1000)
> 
> for(f in colms)
> {
> print("Marker")
> {
> for( i in 1:1000)
> {
> print("perm no.")
> print(se[i])
> {
> peg.no.prm<-sample(peg.no, length(peg.no))
> try(fit5<-lmer(data=modeldf, peg.no.prm~1 + (1|family/f)))
> print(summary(fit5))
> capture.output(fit5, file="testperm5.txt", append=T)
> }}}
> }
> 
> 
> The data files are at:
>>>
>>>
>>> <http://www.4shared.com/file/131980362/460bdafe/Testvcomp10.ht
>>> ml> (excel)
>>> http://www.4shared.com/file/131980512/dc7308b/Testvcomp10.html
>>> (txt)
> ------------------------------------------------------------------
> 
> 
> ----------------------
> A Singh
> Aditi.Singh at bristol.ac.uk
> School of Biological Sciences
> University of Bristol
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
> 
> 
-- 
View this message in context:
http://old.nabble.com/Splitting-massive-output-into-multiple-text-files-tp26429300p26432040.html
Sent from the R help mailing list archive at Nabble.com.