R help - Aug 2012 - Parallel runs of an external executable with snow in local

Hi everyone,

I'm aiming to run an external executable (say filetorun.EXE) in parallel.
The external executable collect needed data from a file, say
"input.txt" and, in turn,generates several output files, say
"output.txt". I need to generate "input.txt", run the
executable and keep "input.txt" and "output.txt". I'm
using Windows 7, R version 2.15.1 (2012-06-22) on RStudio and platform:
i386.pc.mingw32/i386 (32-bit).

My first attempt was a R code which, by using
  System("filetorun.EXE", intern = F, ignore.stdout = F,
          ignore.stderr = F, wait = T, input = NULL,
          show.output.on.console = T, minimized = F, invisible = T))
, ran the executable and kept required files to a conveniently named folder.
After that I changed my previous R script so I could use the function
lapply().This script apparently worked fine.
 
Finally, I tried to parallelize the problem by using snow and parLapply(). The
resulting script looks like this:

## Not run
#
library(snow)cl <- makeCluster(3, type = "SOCK")
clusterExport(cl,list('param.esp','copy.files','for12.template','program.executor'))
parLapply(cl,a.list,a.function))stopCluster(cl)
#
##End not run

Although it runs, the parallelized version is messing up the input parameters to
pass to the executable (see table below, where parameters P1 and P2 are
considered. ".s" comes from the serial code and ".p" from
the parallelized one):
  s r P1.s P2.s P1.p P2.p
1 1 1  1.0 3.00  2.0 3.00
2 2 1  1.5 3.00  2.0 3.75
3 3 1  2.0 3.00  2.0 3.00
4 4 1  1.0 3.75  1.5 3.00
5 5 1  1.5 3.75  1.5 3.00
6 6 1  2.0 3.75  2.0 3.75

My first thought to avoid the described behaviour was creating a temporary file,
say "tmp.id" with id being an identification run number, and copying
"filetorun.EXE" and "Input.txt" to "tmp.id".
However, while doing so, I realised that although running the correct
"filetorun.EXE" copy (i.e., the one in "tmp.id") R looks for
"input.txt" in the work directory.


I've been looking thoroughly for a solution but I got nothing.  

Thanks for any help in advance,


Xavier Portell Canal

PhD candidate
Department of Agri-food engineering,
Universitat Polit?cnica de Catalunya

On 03.08.2012 19:21, Xavier Portell/UPC wrote:
> Hi everyone,
>
> I'm aiming to run an external executable (say filetorun.EXE) in
parallel. The external executable collect needed data from a file, say
"input.txt" and, in turn,generates several output files, say
"output.txt". I need to generate "input.txt", run the
executable and keep "input.txt" and "output.txt". I'm
using Windows 7, R version 2.15.1 (2012-06-22) on RStudio and platform:
i386.pc.mingw32/i386 (32-bit).
>
> My first attempt was a R code which, by using
>    System("filetorun.EXE", intern = F, ignore.stdout = F,
>            ignore.stderr = F, wait = T, input = NULL,
>            show.output.on.console = T, minimized = F, invisible = T))
> , ran the executable and kept required files to a conveniently named
folder. After that I changed my previous R script so I could use the function
lapply().This script apparently worked fine.
>
> Finally, I tried to parallelize the problem by using snow and parLapply().
The resulting script looks like this:
>
> ## Not run
> #
> library(snow)cl <- makeCluster(3, type = "SOCK")
>
clusterExport(cl,list('param.esp','copy.files','for12.template','program.executor'))
> parLapply(cl,a.list,a.function))stopCluster(cl)
> #
> ##End not run
>
> Although it runs, the parallelized version is messing up the input
parameters to pass to the executable (see table below, where parameters P1 and
P2 are considered. ".s" comes from the serial code and ".p"
from the parallelized one):
>    s r P1.s P2.s P1.p P2.p
> 1 1 1  1.0 3.00  2.0 3.00
> 2 2 1  1.5 3.00  2.0 3.75
> 3 3 1  2.0 3.00  2.0 3.00
> 4 4 1  1.0 3.75  1.5 3.00
> 5 5 1  1.5 3.75  1.5 3.00
> 6 6 1  2.0 3.75  2.0 3.75
>
> My first thought to avoid the described behaviour was creating a temporary
file, say "tmp.id" with id being an identification run number, and
copying "filetorun.EXE" and "Input.txt" to
"tmp.id". However, while doing so, I realised that although running
the correct "filetorun.EXE" copy (i.e., the one in "tmp.id")
R looks for "input.txt" in the work directory.

Not sure about the real setup, but you can actually specify the path, 
not only filenames.

Uwe Ligges

>
>
> I've been looking thoroughly for a solution but I got nothing.
>
> Thanks for any help in advance,
>
>
> Xavier Portell Canal
>
> PhD candidate
> Department of Agri-food engineering,
> Universitat Polit?cnica de Catalunya
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>