Read something about parallel processing and how I/O should be done by a
single process.
Suggestion: write a different file from each thread then combine the
results with cat or similar.
Hope it helps
mario
On 04-Jul-11 11:58, Ramzi TEMANNI wrote:> Hi
> I'm processing sequencing data trying to collapsing the locations of
each
> unique sequence and write the results to a file (as storing that in a table
> will require 10GB mem at least)
> so I wrote a function that, given a sequence id, provide the needed line to
> be stored
> library(doMC) # load library
> registerDoMC(12) # assign the Number of CPU
>
>
>
fileConn<-file(paste(fq_file,"_SeqID.txt",sep=""),open =
"at") # open
> connection
>
writeLines(paste("ReadID","Freq","Seq","LOC_UG","Nb_UG_Seq",sep="\t"),
> fileConn) # write header
> foreach(i=1:length(uniq.Seq)) %dopar% # for eqch unique sequence
> {
> writeLines(paste(gettable1(uniq.Seq[i]),collapse=" "),
fileConn) #write
> the the results line
> }
> close(fileConn)
>
> the code excute well, but the problem is that some lines are wired:
> The header and lot of lines are ok :
> ReadID Freq Seq LOC_UG Nb_UG_Seq
> HWI-EA332_0036:5:16:9530:21025#ATGC/1 XXXXXXXXXXXXXXXXXXXX 2
> XXXXX_10130:489:+,XXXXX_10130:489:+ 2
> HWI-EA332_0036:5:117:6674:4940#ATGC/1 XXXXXXXXXXXXXXXXXXXX 1
> XXXXX:432:-,XXXXX:432:- 2
> HWI-EA332_0036:5:62:15592:7375#ATGC/1 XXXXXXXXXXXXXXXXXXXX 2
> XXXXX_22660:253:+,XXXXX_22660:253:+ 2
> HWI-EA332_0036:5:110:14349:8422#ATGC/1 XXXXXXXXXXXXXXXXXXXX 4
> XXXXX_13806:399:+,XXXXX_13806:399:+,XXXXX_27263:481:+,XXXXX_27263:481:+ 4
> other looks wired
> HWI-EA332_0036:5:17:1400ReadID Freq Seq LOC_UG Nb_UG_Seq
> HWI-EA332_0036:5:61:7734:4201ReadID Freq Seq LOC_UG Nb_UG_Seq
> HWI-EA332_0036:5:117:5361:10666#ATGReadID Freq Seq LOC_UG
> Nb_UG_Seq
> HWI-EA332_0036:5:115:7421:20664#ATGC/1 GATCReadID Freq Seq
> LOC_UG Nb_UG_Seq
> HWI-EA332_0036:5:175:95:- 2
> HWI-EA332_0036:5JCVI_35536:444:+ 2
> XXXXXXXXX 1 XXXXX_22484:571:-,XXXXX_22484:571:- 2
>
> Is this due to the fact that one process start to write prior the other has
> finished ?
> Is there a way to solve this problem ?
> Any suggestions would be greatly appreciated.
> Thanks and have a nice day.
>
>
> Best,
> Ramzi TEMANNI
> http://www.linkedin.com/in/ramzitemanni
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
--
Ing. Mario Valle
Data Analysis and Visualization Group | http://www.cscs.ch/~mvalle
Swiss National Supercomputing Centre (CSCS) | Tel: +41 (91) 610.82.60
v. Cantonale Galleria 2, 6928 Manno, Switzerland | Fax: +41 (91) 610.82.82