Backgroundrb devel - Apr 2007 - Problem with blocking workers

Hi Folks,

I found this thread

http://rubyforge.org/pipermail/backgroundrb-devel/2007-March/000790.html

on this mailing list that describes pretty much the same behaviour that I
experience as well.

My problem is, that I have no chance for putting a "sleep 0" in an
iteration because I have no Iteration.

What I?m doing is executing a stored procedure in a sybase db that could
take from few seconds up to hours to complete. The stored procedure gets
executed and spools the result to another table where the user could
access it later. So in my worker I don?t actually iterate over the results
 instead it waits until the stored procedure has finished.

I need to get different jobs with different execution times executed at
the same time. This is how I understand concurrency.

How could I solve my problem? Are concurrent tasks possible anyway?

Thanks & Regards,

Christian Schlaefcke

On Tue, 3 Apr 2007, Christian Schlaefcke wrote:
> Hi Folks,
>
> I found this thread
>
> http://rubyforge.org/pipermail/backgroundrb-devel/2007-March/000790.html
>
> on this mailing list that describes pretty much the same behaviour that I
> experience as well.
>
> My problem is, that I have no chance for putting a "sleep 0" in
an
> iteration because I have no Iteration.
>
> What I?m doing is executing a stored procedure in a sybase db that could
> take from few seconds up to hours to complete. The stored procedure gets
> executed and spools the result to another table where the user could
> access it later. So in my worker I don?t actually iterate over the results
> instead it waits until the stored procedure has finished.
>
> I need to get different jobs with different execution times executed at
> the same time. This is how I understand concurrency.
>
> How could I solve my problem? Are concurrent tasks possible anyway?
>
> Thanks & Regards,
>
> Christian Schlaefcke
Based on my experiences, you will have a difficult time doing what you 
want to do, at least if trying to use backgroundrb. If I were attacking 
this problem, I probably would use a far less elegant approach, that being 
to fork a new child in which the query is run.

Hopefully, others on this list have a better suggestion than I do...

- Marc

It sounds to me like the sybase-ruby driver blocks when being  
called. If something blocks in a ruby thread then it can block all  
other ruby threads. That appears to be what it happening here.

-Ezra


On Apr 3, 2007, at 3:26 AM, Christian Schlaefcke wrote:
> Hi Folks,
>
> I found this thread
>
> http://rubyforge.org/pipermail/backgroundrb-devel/2007-March/ 
> 000790.html
>
> on this mailing list that describes pretty much the same behaviour  
> that I
> experience as well.
>
> My problem is, that I have no chance for putting a "sleep 0" in
an
> iteration because I have no Iteration.
>
> What I?m doing is executing a stored procedure in a sybase db that  
> could
> take from few seconds up to hours to complete. The stored procedure  
> gets
> executed and spools the result to another table where the user could
> access it later. So in my worker I don?t actually iterate over the  
> results
>  instead it waits until the stored procedure has finished.
>
> I need to get different jobs with different execution times  
> executed at
> the same time. This is how I understand concurrency.
>
> How could I solve my problem? Are concurrent tasks possible anyway?
>
> Thanks & Regards,
>
> Christian Schlaefcke
>
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
-- Ezra Zygmuntowicz 
-- Lead Rails Evangelist
-- ez at engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)

Marc Evans schrieb:
> Based on my experiences, you will have a difficult time doing what you 
> want to do, at least if trying to use backgroundrb. If I were 
> attacking this problem, I probably would use a far less elegant 
> approach, that being to fork a new child in which the query is run.
>
> Hopefully, others on this list have a better suggestion than I do...
Could you provide an example for your suggestion? I played around with 
ruby threads before I switched to backgroundrb. I set up a test where 
have two threads that should download a large file from my server. This 
should pretty much simulate what I want to do at work. It seems to me 
that one thread works after the other and not concurrently. I also 
encountered the situation where none of the thread started (race 
condition?).

I would be glad for any hint on this. Hopefully it is possible to solve 
with Backgroundrb...

Regards,

Christian

-------------- next part --------------
A non-text attachment was scrubbed...
Name: cschlaefcke.vcf
Type: text/x-vcard
Size: 368 bytes
Desc: not available
Url :
http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20070403/75a60675/attachment-0001.vcf

On Tue, 3 Apr 2007, Christian Schlaefcke wrote:
> Marc Evans schrieb:
>> Based on my experiences, you will have a difficult time doing what you
want
>> to do, at least if trying to use backgroundrb. If I were attacking this
>> problem, I probably would use a far less elegant approach, that being
to
>> fork a new child in which the query is run.
>> 
>> Hopefully, others on this list have a better suggestion than I do...
> Could you provide an example for your suggestion? I played around with ruby
> threads before I switched to backgroundrb. I set up a test where have two 
> threads that should download a large file from my server. This should
pretty
> much simulate what I want to do at work. It seems to me that one thread
works
> after the other and not concurrently. I also encountered the situation
where
> none of the thread started (race condition?).
>
> I would be glad for any hint on this. Hopefully it is possible to solve
with
> Backgroundrb...
My suggestion was to use Process.fork, e.g. avoid the ruby thread issue. 
It''s much heavier weight, and requires that you re-invent status
tracking
and related mechanisms. That said, I don''t believe that you will
encounter
the blocking problems observed in thread-style code.

- Marc