R devel - Jan 2007 - How to execute R scripts simultaneously from multiple threads

Hi All,

My problem is about parallel execution of R-scripts. My platform is linux.

A program that is written in C needs to execute multiple R-scripts 
simultaneously. The C program makes use of multi-threading. Each thread 
must initiate the execution of one script. Performance is very important.

Appearantly the R C-API does not provide a mechanism for parallel 
execution..

It is preferred that the solution is not based on multi-processing (like 
C/S), because that would introduce IPC overhead.

Hopefully some thread-safe (single-proces) solution is readily 
available, written in C.

What is the best solution to do this?

(If there is no single-process solution, what is the alternative?)

Regards,
Erik.
-- 
Shannon's Observation:
	Nothing is so frustrating as a bad situation that is beginning to
	improve.

Short Answer: There isn't one.

On 1/3/07, Erik van Zijst <r at erik.prutser.cx>
wrote:
> Hi All,
>
> My problem is about parallel execution of R-scripts. My platform is linux.
>
> A program that is written in C needs to execute multiple R-scripts
> simultaneously. The C program makes use of multi-threading. Each thread
> must initiate the execution of one script. Performance is very important.
>
> Appearantly the R C-API does not provide a mechanism for parallel
> execution..
>
> It is preferred that the solution is not based on multi-processing (like
> C/S), because that would introduce IPC overhead.
>
> Hopefully some thread-safe (single-proces) solution is readily
> available, written in C.
>
> What is the best solution to do this?
>
> (If there is no single-process solution, what is the alternative?)
>
> Regards,
> Erik.
> --
> Shannon's Observation:
>         Nothing is so frustrating as a bad situation that is beginning to
>         improve.
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

-- 
Byron Ellis (byron.ellis at gmail.com)
"Oook" -- The Librarian

On Wednesday 03 January 2007 3:47 am, Erik van Zijst
wrote:
> Hi All,
>
> My problem is about parallel execution of R-scripts. My platform is linux.
>
> A program that is written in C needs to execute multiple R-scripts
> simultaneously. The C program makes use of multi-threading. Each thread
> must initiate the execution of one script. Performance is very important.
>
> Appearantly the R C-API does not provide a mechanism for parallel
> execution..
>
> It is preferred that the solution is not based on multi-processing (like
> C/S), because that would introduce IPC overhead.
One thing to keep in mind is that IPC is very fast in Linux. So unless you are 
making lots of calls to really tiny functions this should not be an issue.

What can be an issue is the overhead of starting a new R process. In which 
case you can make some helper processes that do the same thing you wanted 
from a multi-thread one and just pass the data around.

                             best

                                  Vladimir Dergachev
>
> Hopefully some thread-safe (single-proces) solution is readily
> available, written in C.
>
> What is the best solution to do this?
>
> (If there is no single-process solution, what is the alternative?)
>
> Regards,
> Erik.