R devel - Oct 2012 - concurrent requests (Rook, but I think the question is more general)

This question involves Rook, but I think the answer will be general 
enough that it pays to post here. At any rate, I don't know enough to 
know whether this is a Rook only issue or a general R issue.

Here's what I'd like to do (and indeed, have code that should do this):

1. Start R, Rook
2. Start an analysis via a HTTP request to Rook. This analysis uses 
.Call() to some compiled C code, if that matters. The C code calls a 
callback function to update a variable with its progress.
3. While the analysis is happening, use Rook to obtain current status 
with an HTTP request

The problem is that once the analysis starts, Rook does not respond to 
requests. All of the status requests to Rook pile up, and then are 
answered when the analysis (step 2) is done. Here is some example code 
to demonstrate what the issue:

##########

library(Rook)
s <- Rhttpd$new()
s$add(
   name="pingpong",
   app=Rook::URLMap$new(
     '/ping' = function(env){
       req <- Rook::Request$new(env)
       res <- Rook::Response$new()
       res$write('This is ping.')
       Sys.sleep(20)
       res$finish()
     },
     '/pong' = function(env){
       req <- Rook::Request$new(env)
       res <- Rook::Response$new()
       res$write("This is pong.")
       res$finish()
     },
     '/?' = function(env){
       req <- Rook::Request$new(env)
       res <- Rook::Response$new()
       res$redirect(req$to_url('/pong'))
       res$finish()
     }
   )
)

s$start(quiet=TRUE)
s$browse('pingpong')

#############################

If you request /ping, R runs Sys.sleep() for 20 seconds. This is where 
my .Call() statement would be. While the .Call() (Sys.sleep()) function 
is doing its thing, I need to get Rook to respond on /pong (which would 
simply respond with the progress), but if you run this code, request 
/ping, then immediately request /pong, you'll see that the /pong request 
will not be answered until the Sys.sleep() is done.

Of course, for a progress report to be useful, the requests have to be 
answered immediately. Is this a Rook issue, or an R issue? Or am I 
asking something unreasonable?

On Wed, Oct 24, 2012 at 11:13 AM, Richard D. Morey <r.d.morey at rug.nl>
wrote:
> This question involves Rook, but I think the answer will be general enough
> that it pays to post here. At any rate, I don't know enough to know
whether
> this is a Rook only issue or a general R issue.
>
> Here's what I'd like to do (and indeed, have code that should do
this):
>
> 1. Start R, Rook
> 2. Start an analysis via a HTTP request to Rook. This analysis uses .Call()
> to some compiled C code, if that matters. The C code calls a callback
> function to update a variable with its progress.
> 3. While the analysis is happening, use Rook to obtain current status with
> an HTTP request
>
> The problem is that once the analysis starts, Rook does not respond to
> requests. All of the status requests to Rook pile up, and then are answered
> when the analysis (step 2) is done. Here is some example code to
demonstrate
> what the issue:
>
> ##########
>
> library(Rook)
> s <- Rhttpd$new()
> s$add(
>   name="pingpong",
>   app=Rook::URLMap$new(
>     '/ping' = function(env){
>       req <- Rook::Request$new(env)
>       res <- Rook::Response$new()
>       res$write('This is ping.')
>       Sys.sleep(20)
>       res$finish()
>     },
>     '/pong' = function(env){
>       req <- Rook::Request$new(env)
>       res <- Rook::Response$new()
>       res$write("This is pong.")
>       res$finish()
>     },
>     '/?' = function(env){
>       req <- Rook::Request$new(env)
>       res <- Rook::Response$new()
>       res$redirect(req$to_url('/pong'))
>       res$finish()
>     }
>   )
> )
>
> s$start(quiet=TRUE)
> s$browse('pingpong')
>
> #############################
>
> If you request /ping, R runs Sys.sleep() for 20 seconds. This is where my
> .Call() statement would be. While the .Call() (Sys.sleep()) function is
> doing its thing, I need to get Rook to respond on /pong (which would simply
> respond with the progress), but if you run this code, request /ping, then
> immediately request /pong, you'll see that the /pong request will not
be
> answered until the Sys.sleep() is done.
>
> Of course, for a progress report to be useful, the requests have to be
> answered immediately. Is this a Rook issue, or an R issue? Or am I asking
> something unreasonable?
One answer would be to start an Rserve instance on your local machine.
When your web app initiates processing, it actually starts the
long-running task on the server with RS.eval(wait=FALSE). See ?RCC
with the RS.client package loaded.
Then when you check for task completion, call RS.collect () with a
short timeout, and if it has something for you it will give it to you.

That doesn't give you a numeric progress report, but perhaps if your
long-running task writes its status somewhere (to a file?) the
progress-checking task could look there as well.

Dan

>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

On Oct 24, 2012, at 2:13 PM, Richard D. Morey wrote:
> This question involves Rook, but I think the answer will be general enough
that it pays to post here. At any rate, I don't know enough to know whether
this is a Rook only issue or a general R issue.
> 
> Here's what I'd like to do (and indeed, have code that should do
this):
> 
> 1. Start R, Rook
> 2. Start an analysis via a HTTP request to Rook. This analysis uses .Call()
to some compiled C code, if that matters. The C code calls a callback function
to update a variable with its progress.
> 3. While the analysis is happening, use Rook to obtain current status with
an HTTP request
> 
You can't. R doesn't support threading so it's simply not possible
to have an asynchronous eval. The R HTTP server works by simply enqueuing an
eval to run while R is idle, it can't do that if R is busy. (Note that the
HTTP server was *only* designed for the internal help).

What you can do is have your C code start another thread that reports the
progress when asked e.g. on a socket, but that thread is not allowed to call any
R API so you want that progress to be entirely in your C code.

Note that if your C code is robust enough, it can call R_CheckUserInterrupt() to
allow external events to happen, but a) your C code must in that case be
prepared for early termination (clean up memory etc.) and b) I don't
remember if the httpd is allowed to run during interrupt check on all platforms
- you may want to check that first.

Cheers,
Simon



> The problem is that once the analysis starts, Rook does not respond to
requests. All of the status requests to Rook pile up, and then are answered when
the analysis (step 2) is done. Here is some example code to demonstrate what the
issue:
> 
> ##########
> 
> library(Rook)
> s <- Rhttpd$new()
> s$add(
>  name="pingpong",
>  app=Rook::URLMap$new(
>    '/ping' = function(env){
>      req <- Rook::Request$new(env)
>      res <- Rook::Response$new()
>      res$write('This is ping.')
>      Sys.sleep(20)
>      res$finish()
>    },
>    '/pong' = function(env){
>      req <- Rook::Request$new(env)
>      res <- Rook::Response$new()
>      res$write("This is pong.")
>      res$finish()
>    },
>    '/?' = function(env){
>      req <- Rook::Request$new(env)
>      res <- Rook::Response$new()
>      res$redirect(req$to_url('/pong'))
>      res$finish()
>    }
>  )
> )
> 
> s$start(quiet=TRUE)
> s$browse('pingpong')
> 
> #############################
> 
> If you request /ping, R runs Sys.sleep() for 20 seconds. This is where my
.Call() statement would be. While the .Call() (Sys.sleep()) function is doing
its thing, I need to get Rook to respond on /pong (which would simply respond
with the progress), but if you run this code, request /ping, then immediately
request /pong, you'll see that the /pong request will not be answered until
the Sys.sleep() is done.
> 
> Of course, for a progress report to be useful, the requests have to be
answered immediately. Is this a Rook issue, or an R issue? Or am I asking
something unreasonable?
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
> 
>