Rails - Dec 2007 - BackgrounDRb 1.0 pre-release available now

Hi Folks,

We are glad to announce shiny new release of  BackgrounDRb, which will
soon become 1.0.

A quick summary of changes:

- BackgrounDRb is no londer DRb, its based on event driven network
programming library packet ( http://www.packet.googlecode.com ) .

- Since we moved to packet, many nasty thread issues, result hash
corruption issues are totally gone. Lots of work has went in
  making scheduler rock solid stable.

- Each worker, still runs in its own process, but each worker has a
event loop of its own and all the events are triggered by the internal
  reactor loop. In a nutshell, you are not encouraged to use threads
in your workers now. All the workers are already concurrent, but you
  are encouraged to use co-operative multitasking, rather than
pre-emptive. A simple example is,

  For implement something like progress bar in old version of bdrb, you would:
    - start your processing a thread (so as your worker can receive
further request from rails ) and have a instance
      variable ( protected by mutex ) which is updated on progress and
can be send to rails.

  With new backgroundrb, progrss bar would be:
    - process your damn request and just use register_status() to
register status of your worker. Just because
      you are doing some processing won''t mean that your worker will
block. It can still receive requests from rails.


- Now, you can schedule mulitple methods with their own triggers.

- Inside each worker, you can start tcp server or connect to a
external server. Two important methods available in all workers are:

   start_server("localhost",port,ModuleName)
   connect("localhost",port,ModuleName)

  Connected client or outgoing connection would be integrated with
Event Loop and you can process requests from these guys
  asynchronously. This mouse trap can allow you to build truly
distributed workers across your network.


The detailed list of changes can be found here:

http://backgroundrb.rubyforge.org/

Please give it a try and let me know if you found any bugs.


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

On Dec 10, 2007, at 6:47 PM, hemant wrote:
> Hi Folks,
>
> We are glad to announce shiny new release of  BackgrounDRb, which will
> soon become 1.0.
>
> A quick summary of changes:
>
> - BackgrounDRb is no londer DRb, its based on event driven network
> programming library packet ( http://www.packet.googlecode.com ) .
>
> - Since we moved to packet, many nasty thread issues, result hash
> corruption issues are totally gone. Lots of work has went in
>  making scheduler rock solid stable.
>
> - Each worker, still runs in its own process, but each worker has a
> event loop of its own and all the events are triggered by the internal
>  reactor loop. In a nutshell, you are not encouraged to use threads
> in your workers now. All the workers are already concurrent, but you
>  are encouraged to use co-operative multitasking, rather than
> pre-emptive. A simple example is,
>
>  For implement something like progress bar in old version of bdrb,  
> you would:
>    - start your processing a thread (so as your worker can receive
> further request from rails ) and have a instance
>      variable ( protected by mutex ) which is updated on progress and
> can be send to rails.
>
>  With new backgroundrb, progrss bar would be:
>    - process your damn request and just use register_status() to
> register status of your worker. Just because
>      you are doing some processing won''t mean that your worker
will
> block. It can still receive requests from rails.
>
>
> - Now, you can schedule mulitple methods with their own triggers.
>
> - Inside each worker, you can start tcp server or connect to a
> external server. Two important methods available in all workers are:
>
>   start_server("localhost",port,ModuleName)
>   connect("localhost",port,ModuleName)
>
>  Connected client or outgoing connection would be integrated with
> Event Loop and you can process requests from these guys
>  asynchronously. This mouse trap can allow you to build truly
> distributed workers across your network.
>
>
> The detailed list of changes can be found here:
>
> http://backgroundrb.rubyforge.org/
>
> Please give it a try and let me know if you found any bugs.
Thanks for all your hard work on this recently.

At one point with the old version it was fairly straight-forward to  
test workers, but that broke at one point.  Could you give any  
pointers writing tests for workers in the new version?

Thanks,
Brandon
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url :
http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20071210/22882edb/attachment-0001.bin

Hi there,

On Dec 11, 2007 12:47 AM, hemant <gethemant at gmail.com>
wrote:
> Hi Folks,
>
> We are glad to announce shiny new release of  BackgrounDRb, which will
> soon become 1.0.
>
> A quick summary of changes:
>
> - BackgrounDRb is no londer DRb, its based on event driven network
> programming library packet ( http://www.packet.googlecode.com ) .
>
> - Since we moved to packet, many nasty thread issues, result hash
> corruption issues are totally gone. Lots of work has went in
>   making scheduler rock solid stable.

Does this means that slave/daemons are not the dependency anymore?

> - Each worker, still runs in its own process, but each worker has a
> event loop of its own and all the events are triggered by the internal
>   reactor loop. In a nutshell, you are not encouraged to use threads
> in your workers now.
By ''not encouraged'' do you mean that 1.0 is not supporting
multiple
threads in the worker or just as a general guidance?

Could you please comment, how would you approach the following
scenario with 1.0. Currently, we have a worker that creates threads
that process financial payment transactions. An http request sends
several 10s or 100s payment transaction records. They are handled by
the single worker instance. Within the worker there is a pool of
threads created that is calculated based on the number of
transactions. For example for 200 transactions there will be 20
threads where each thread handles 10 requests in a squence. Each
transaction takes about 3-5 seconds, so our throughput is
significantly improved by internal worker parallelization with a
thread pool. The worker periodically updates custom backgroundjob
databse record, so that following ajax request from the client can
read the status of the worker process. The job is identified with the
worker key that is stored in the session.
> All the workers are already concurrent, but you
>   are encouraged to use co-operative multitasking, rather than
> pre-emptive. A simple example is,
>
>   For implement something like progress bar in old version of bdrb, you
would:
>     - start your processing a thread (so as your worker can receive
> further request from rails ) and have a instance
>       variable ( protected by mutex ) which is updated on progress and
> can be send to rails.
>
>   With new backgroundrb, progrss bar would be:
>     - process your damn request and just use register_status() to
> register status of your worker. Just because
>       you are doing some processing won''t mean that your worker
will
> block. It can still receive requests from rails.
How this works with fastcgi or multiple mongrel based engines where it
is not guaranteed to hit the same process with the next request? We
are using custom database tables and code for sharing the status
information now but I was wandering whether the plumbing includes
something to address this.


thanks,
emil

Hi

On Tue, 2007-12-11 at 10:53 +0100, Emil Marceta wrote:
> Does this means that slave/daemons are not the dependency anymore?
Yes, its gone. bdrb no longer depends on slave and daemons.
> By ''not encouraged'' do you mean that 1.0 is not
supporting multiple
> threads in the worker or just as a general guidance?
> 
> Could you please comment, how would you approach the following
> scenario with 1.0. Currently, we have a worker that creates threads
> that process financial payment transactions. An http request sends
> several 10s or 100s payment transaction records. They are handled by
> the single worker instance. Within the worker there is a pool of
> threads created that is calculated based on the number of
> transactions. For example for 200 transactions there will be 20
> threads where each thread handles 10 requests in a squence. Each
> transaction takes about 3-5 seconds, so our throughput is
> significantly improved by internal worker parallelization with a
> thread pool. The worker periodically updates custom backgroundjob
> databse record, so that following ajax request from the client can
> read the status of the worker process. The job is identified with the
> worker key that is stored in the session.
Its not encouraged, thats all. You can still have threads in your
workers. However, I am planning to add thread pool feature in bdrb
itself, that should simplify things a bit.

Also ideally, when using EventDriven network programming, you want all
your sockets within select loop for efficiency. So, you wouldn''t need
any damn threads, if you can use a HTTP handler that works in Evented
manner. What i mean to say is, you don''t do this:

a = Net::HTTP.get("http://www.google.com")

but you do,

Backgroundrb::HTTP.open("http://www.google.com") do |data|
process_data(data) 
end

What I am trying to illustrate is, when you ask to open, google.com 
page, evented model allows you to attach callback ( the block in this
case ), which will be called when data arrives from google.com, rather
than waiting for it in a thread. So, BackgrounDRb::HTTP.open() returns
immediately. And you are concurrent as hell.

But this is not possible, because if you are charging cards, then you
are probably using ActiveMerchant which is using Net::HTTP and which
blocks when you make request. But trust me, writing a simple http client
is not that difficult, there is already connect() available in all
workers. 

> How this works with fastcgi or multiple mongrel based engines where it
> is not guaranteed to hit the same process with the next request? We
> are using custom database tables and code for sharing the status
> information now but I was wandering whether the plumbing includes
> something to address this.
Thats no problem at all, BackgrounDRb is a TCP server, so if you have
followed the README file, no matter from which machine, you are making
the request if you are specifying worker X, then its guaranteed to hit
the same worker(with optional job_key if you are starting your worker
dynamically)

> At one point with the old version it was fairly straight-forward to  
> test workers, but that broke at one point.  Could you give any  
> pointers writing tests for workers in the new version?
> 
Hi Brandon,

update your bdrb copy from svn and run rake backgroundrb:setup and you
should have a RAILS_ROOT/test/bdrb_test_helper.rb file.

Now, all your worker test cases can go in RAILS_ROOT/test/unit
directory, just make sure that you require bdrb_test_helper file, and
you can write test cases.

For example:

require File.join(File.dirname(__FILE__) + "/../bdrb_test_helper")
require "god_worker"

context "When god worker starts" do
  setup do
    god_worker = GodWorker.new
  end
end

I hope this helps.

Hemant, this looks great.  Could one use BackgroundRb to have workers
interact programatically with a remote telnet service?  Or would I
simply start a worker that does this interaction via a
shell/spawn/telnet/expect...

Great doco too, thanks.

George

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

On Tue, 2007-12-11 at 23:08 +1100, George Bray wrote:
> Hemant, this looks great.  Could one use BackgroundRb to have workers
> interact programatically with a remote telnet service?  Or would I
> simply start a worker that does this interaction via a
> shell/spawn/telnet/expect...
Sure as hell.. with any tcp service actually in a evented manner.
However, that area is not polished ( no one ever asked. :) )

@Hemant :

Has this been tested on Windows? If so, are there known issues? Previous
versions did not work on Windows, although the original version did.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

On Dec 11, 2007 11:50 AM, hemant kumar <gethemant at gmail.com>
wrote:
> Hi
>
> On Tue, 2007-12-11 at 10:53 +0100, Emil Marceta wrote:
>
> > Does this means that slave/daemons are not the dependency anymore?
>
> Yes, its gone. bdrb no longer depends on slave and daemons.
>
> > By ''not encouraged'' do you mean that 1.0 is not
supporting multiple
> > threads in the worker or just as a general guidance?
> >
> > Could you please comment, how would you approach the following
> > scenario with 1.0. Currently, we have a worker that creates threads
> > that process financial payment transactions. An http request sends
> > several 10s or 100s payment transaction records. They are handled by
> > the single worker instance. Within the worker there is a pool of
> > threads created that is calculated based on the number of
> > transactions. For example for 200 transactions there will be 20
> > threads where each thread handles 10 requests in a squence. Each
> > transaction takes about 3-5 seconds, so our throughput is
> > significantly improved by internal worker parallelization with a
> > thread pool. The worker periodically updates custom backgroundjob
> > databse record, so that following ajax request from the client can
> > read the status of the worker process. The job is identified with the
> > worker key that is stored in the session.
>
> Its not encouraged, thats all. You can still have threads in your
> workers.
Thanks for clarifying.
snip ...
> Also ideally, when using EventDriven network programming, you want all
> your sockets within select loop for efficiency. So, you wouldn''t
need
> any damn threads, if you can use a HTTP handler that works in Evented
> manner. What i mean to say is, you don''t do this:
>
> a = Net::HTTP.get("http://www.google.com")
>
> but you do,
>
> Backgroundrb::HTTP.open("http://www.google.com") do |data|
> process_data(data)
> end
>
> What I am trying to illustrate is, when you ask to open, google.com
> page, evented model allows you to attach callback ( the block in this
> case ), which will be called when data arrives from google.com, rather
> than waiting for it in a thread. So, BackgrounDRb::HTTP.open() returns
> immediately. And you are concurrent as hell.
>
> But this is not possible, because if you are charging cards, then you
> are probably using ActiveMerchant which is using Net::HTTP and which
> blocks when you make request. But trust me, writing a simple http client
> is not that difficult, there is already connect() available in all
> workers.
We are actually on of the ActiveMerchant providers (E-Xact), so
strictly we are talking what is actually behind ActiveMerchant. There
are many protocols involved in financial networks, depending where the
transaction is routed. We are very familiar with Reactor engines and
patterns you are advocating, and they work great, especially in
uniform scenarios without throttling, sequencing etc. In our case, I
don''t see a clear gain I''m afraid. While a thread pool was
done in
no-time and is dead simple maintain, test etc.
>
>
> > How this works with fastcgi or multiple mongrel based engines where it
> > is not guaranteed to hit the same process with the next request? We
> > are using custom database tables and code for sharing the status
> > information now but I was wandering whether the plumbing includes
> > something to address this.
>
> Thats no problem at all, BackgrounDRb is a TCP server, so if you have
> followed the README file, no matter from which machine, you are making
> the request if you are specifying worker X, then its guaranteed to hit
> the same worker(with optional job_key if you are starting your worker
> dynamically)
Our Rails cluster runs bdrb on each Rails server and uses domain
sockets. This to avoid a single point of failure and have uniform
architecture. Would that work too? That is, does bdrb now works sort
of like memcache where each server knows of every other instance? But
even with that in place, in fastcgi for example, fastcgi processor may
recycle the Rails process where callback has been registered.


thanks,
emil

Hi

On Tue, 2007-12-11 at 14:52 +0100, Emil Marceta wrote:
> We are actually on of the ActiveMerchant providers (E-Xact), so
> strictly we are talking what is actually behind ActiveMerchant. There
> are many protocols involved in financial networks, depending where the
> transaction is routed. We are very familiar with Reactor engines and
> patterns you are advocating, and they work great, especially in
> uniform scenarios without throttling, sequencing etc. In our case, I
> don''t see a clear gain I''m afraid. While a thread pool
was done in
> no-time and is dead simple maintain, test etc.

Cool. You can use existing approach provided you handle your threads
with as much care. I will get back to this in sometime. There are other
ways also, that I am looking. For example: co-routines ( on top of
fibers ) from Ruby1.9. Just watch bdrb mailing list, or submit some
patch. As i guess, you guys are already running somewhat customized
version of bdrb.

> Our Rails cluster runs bdrb on each Rails server and uses domain
> sockets. This to avoid a single point of failure and have uniform
> architecture. Would that work too? That is, does bdrb now works sort
> of like memcache where each server knows of every other instance? But
> even with that in place, in fastcgi for example, fastcgi processor may
> recycle the Rails process where callback has been registered.
Hmm, this is cool. So, how did you handle this situation earlier?
Prolly, what you can do is, have bdrb instances running on each cluster
and have cluster specific backgroundrb configuration file. So as,
requests from  mongrels running on cluster1 will be served by bdrb
running on cluster1 only, and update some db/memcache key to indicate
it, so as even if next time request goes to another worker on another
machine, you know the state.
Again, I would love any patch, ideas from you and I am myself working on
something like this, which would avoid logging to db and stuff.



-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

On Tue, 2007-12-11 at 07:45 -0600, Brian Hogan wrote:
> @Hemant :
>  
> Has this been tested on Windows? If so, are there known issues?
> Previous versions did not work on Windows, although the original
> version did.

No, it won''t work on Windows. Even when I removed "slave",
still we need
unix domain sockets for internal communication, which is not available
on windows.


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

@Hemant:

Thank you. That''s what I suspected.

On Dec 11, 2007 9:00 AM, hemant kumar
<gethemant-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
>
> On Tue, 2007-12-11 at 07:45 -0600, Brian Hogan wrote:
> > @Hemant :
> >
> > Has this been tested on Windows? If so, are there known issues?
> > Previous versions did not work on Windows, although the original
> > version did.
>
>
> No, it won''t work on Windows. Even when I removed
"slave", still we need
> unix domain sockets for internal communication, which is not available
> on windows.
>
>
> --
> Let them talk of their oriental summer climes of everlasting
> conservatories; give me the privilege of making my own summer with my
> own coals.
>
> http://gnufied.org
>
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi,

Looking forward to a chance to use this library. Thanks for the work!

On 10-Dec-07, at 6:47 PM, hemant wrote:
> - BackgrounDRb is no londer DRb, its based on event driven network
> programming library packet ( http://www.packet.googlecode.com ) .

How does this affect the licensing of BackgrounDRb (not to mention the  
name of the project :-)? The packet library is GPLv2 (the url doesn''t  
have the leading www by the way), while BackgrounDRb is dual licensed  
with the Ruby License or an MIT license.

Cheers,
Bob

----
Bob Hutchison                  -- tumblelog at http://www.recursive.ca/so/
Recursive Design Inc.          -- weblog at http://www.recursive.ca/hutch
http://www.recursive.ca/       -- works on
http://www.raconteur.info/cms-for-static-content/home/

Hi,

Looking forward to a chance to use this library. Thanks for the work!

On 10-Dec-07, at 6:47 PM, hemant wrote:
> - BackgrounDRb is no londer DRb, its based on event driven network
> programming library packet ( http://www.packet.googlecode.com ) .

How does this affect the licensing of BackgrounDRb (not to mention the  
name of the project :-)? The packet library is GPLv2 (the url doesn''t  
have the leading www by the way), while BackgrounDRb is dual licensed  
with the Ruby License or an MIT license.

Cheers,
Bob


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi

On Wed, 2007-12-12 at 11:35 -0500, Bob Hutchison wrote:
> How does this affect the licensing of BackgrounDRb (not to mention the  
> name of the project :-)? The packet library is GPLv2 (the url
doesn''t
> have the leading www by the way), while BackgrounDRb is dual licensed  
> with the Ruby License or an MIT license.
Damn I realized it after posting the message. But then thought
"packet"
may be irrelevant anyways ( to rails guys i mean ) 

Regarding license issues, since packet is dual licensed under GPL2 and
Ruby, you can take shit from packet and embed in your app and forget
that its under GPL2, since Ruby license allows you do that. There is a
clause from Ruby license that says:

"place your modifications in the Public Domain or otherwise
 make them Freely Available, such as by posting said
 modifications to Usenet or an equivalent medium, or by allowing
 the author to include your modifications in the software."

So, I guess its ok to have that.


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

Question: anyone know the ''latest'' version that does work with
windows
(of backgroundrb?)
Thanks!
-Roger
Brian Hogan wrote:
> @Hemant :
> 
> Has this been tested on Windows? If so, are there known issues? Previous
> versions did not work on Windows, although the original version did.
-- 
Posted via http://www.ruby-forum.com/.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

svn checkout svn://rubyforge.org/var/svn/backgroundrb
> Question: anyone know the ''latest'' version that does work
with windows
> (of backgroundrb?)
> Thanks!
> -Roger
-- 
Posted via http://www.ruby-forum.com/.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

> - BackgrounDRb is no londer DRb, its based on event driven network
> programming library packet ( http://www.packet.googlecode.com ) .
http://rubyforge.org/projects/packet/

Question: how does this compare to EventMachine?
> - Since we moved to packet, many nasty thread issues, result hash
> corruption issues are totally gone. Lots of work has went in
>   making scheduler rock solid stable.
-- 
Posted via http://www.ruby-forum.com/.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

On Dec 29, 2007 1:58 AM, Roger Pack
<rails-mailing-list-ARtvInVfO7ksV2N9l4h3zg@public.gmane.org>
wrote:
>
>
> > - BackgrounDRb is no londer DRb, its based on event driven network
> > programming library packet ( http://www.packet.googlecode.com ) .
> http://rubyforge.org/projects/packet/
>
> Question: how does this compare to EventMachine?
Sorry about wrong link, correct one is: http://packet.googlecode.com

Well, i think one of the strengths of packet is, it lets you write
tightly integrated workers with master process. So, this way, you can
offload blocking tasks to these workers, which will run parallely and
keep processing further requests in master.

And its pure ruby.


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

>> Question: how does this compare to EventMachine?
> 
> Sorry about wrong link, correct one is: http://packet.googlecode.com
> 
> Well, i think one of the strengths of packet is, it lets you write
> tightly integrated workers with master process. So, this way, you can
> offload blocking tasks to these workers, which will run parallely and
> keep processing further requests in master.
> 
> And its pure ruby.
Sounds cool.  I think EM also does pure ruby these days--seems like a 
trend :)
Now if I can just figure out which is the latest version of 
acts_as_ferret that runs on win32... :)
Thanks for any help.
-Roger
-- 
Posted via http://www.ruby-forum.com/.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Ruby on Rails: Talk" group.
To post to this group, send email to
rubyonrails-talk-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
To unsubscribe from this group, send email to
rubyonrails-talk-unsubscribe-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi Hemant,

Sorry for referring to an old thread, but I''m confused about
appropriate
times to use thread_pool vs putting sockets in the select loop (or if that
has been deprecated).

In your thread_pool example, you have this:

def fetch_url(url)
  puts "fetching url #{url}"
  thread_pool.defer(url) do |url|
    begin
      data = Net::HTTP.get(url,''/'')
      File.open("#{RAILS_ROOT}/log/pages.txt" ,"w") do |fl|
        fl.puts(data)
      end
   rescue
     logger.info "Error downloading page"
   end
 end
end

But in this thread (from Dec 11) you mentioned that it is much better to put
connections in the select loop via Backgroundrb::HTTP.open to stay with the
evented paradigm instead of directly calling Net::HTTP.  In my case, I want
to open up a bunch of lightweight connections to various servers, but
I''d
like to have at most 20 open at a time.  In this case, would it make sense
to use a thread_pool of size 20 to manage the size of the connection pool,
but also use Backgroundrb:: HTTP.open for better socket concurrency?  Just
trying to get this all straight.

Thanks!

stevie


On 12/11/07, hemant kumar <gethemant at gmail.com>
wrote:
>
> Hi
>
> On Tue, 2007-12-11 at 10:53 +0100, Emil Marceta wrote:
>
> > Does this means that slave/daemons are not the dependency anymore?
>
> Yes, its gone. bdrb no longer depends on slave and daemons.
>
> > By ''not encouraged'' do you mean that 1.0 is not
supporting multiple
> > threads in the worker or just as a general guidance?
> >
> > Could you please comment, how would you approach the following
> > scenario with 1.0. Currently, we have a worker that creates threads
> > that process financial payment transactions. An http request sends
> > several 10s or 100s payment transaction records. They are handled by
> > the single worker instance. Within the worker there is a pool of
> > threads created that is calculated based on the number of
> > transactions. For example for 200 transactions there will be 20
> > threads where each thread handles 10 requests in a squence. Each
> > transaction takes about 3-5 seconds, so our throughput is
> > significantly improved by internal worker parallelization with a
> > thread pool. The worker periodically updates custom backgroundjob
> > databse record, so that following ajax request from the client can
> > read the status of the worker process. The job is identified with the
> > worker key that is stored in the session.
>
> Its not encouraged, thats all. You can still have threads in your
> workers. However, I am planning to add thread pool feature in bdrb
> itself, that should simplify things a bit.
>
> Also ideally, when using EventDriven network programming, you want all
> your sockets within select loop for efficiency. So, you wouldn''t
need
> any damn threads, if you can use a HTTP handler that works in Evented
> manner. What i mean to say is, you don''t do this:
>
> a = Net::HTTP.get("http://www.google.com")
>
> but you do,
>
> Backgroundrb::HTTP.open("http://www.google.com") do |data|
> process_data(data)
> end
>
> What I am trying to illustrate is, when you ask to open, google.com
> page, evented model allows you to attach callback ( the block in this
> case ), which will be called when data arrives from google.com, rather
> than waiting for it in a thread. So, BackgrounDRb::HTTP.open() returns
> immediately. And you are concurrent as hell.
>
> But this is not possible, because if you are charging cards, then you
> are probably using ActiveMerchant which is using Net::HTTP and which
> blocks when you make request. But trust me, writing a simple http client
> is not that difficult, there is already connect() available in all
> workers.
>
>
> > How this works with fastcgi or multiple mongrel based engines where it
> > is not guaranteed to hit the same process with the next request? We
> > are using custom database tables and code for sharing the status
> > information now but I was wandering whether the plumbing includes
> > something to address this.
>
> Thats no problem at all, BackgrounDRb is a TCP server, so if you have
> followed the README file, no matter from which machine, you are making
> the request if you are specifying worker X, then its guaranteed to hit
> the same worker(with optional job_key if you are starting your worker
> dynamically)
>
>
>
> _______________________________________________
> Backgroundrb-devel mailing list
> Backgroundrb-devel at rubyforge.org
> http://rubyforge.org/mailman/listinfo/backgroundrb-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/backgroundrb-devel/attachments/20080109/cc7a4947/attachment-0001.html

Hi Stevie,

The problem is BackgrounDRb::HTTP doesn''t exist. I just tried to
explain that, if you are really concerned about performance you can
implement HTTP client in a non blocking manner and you won''t need
threads. For example, EventMachine guys have already implemented HTTP
client over evented sockets. You can use their code, I guess.

But its not easy task, unless BackgrounDRb itself provides this, so
for the time being go with threads.




On Jan 10, 2008 12:47 AM, Stevie Clifton <stevie at slowbicycle.com>
wrote:
>
> Hi Hemant,
>
> Sorry for referring to an old thread, but I''m confused about
appropriate
> times to use thread_pool vs putting sockets in the select loop (or if that
> has been deprecated).
>
> In your thread_pool example, you have this:
>
> def fetch_url(url)
>   puts "fetching url #{url}"
>   thread_pool.defer(url) do |url|
>     begin
>       data = Net::HTTP.get(url,''/'')
>       File.open("#{RAILS_ROOT}/log/pages.txt" ,"w") do
|fl|
>         fl.puts(data)
>       end
>    rescue
>      logger.info "Error downloading page"
>    end
>  end
> end
>
> But in this thread (from Dec 11) you mentioned that it is much better to
put
> connections in the select loop via Backgroundrb::HTTP.open to stay with the
> evented paradigm instead of directly calling Net::HTTP.  In my case, I want
> to open up a bunch of lightweight connections to various servers, but
I''d
> like to have at most 20 open at a time.  In this case, would it make sense
> to use a thread_pool of size 20 to manage the size of the connection pool,
> but also use Backgroundrb:: HTTP.open for better socket concurrency?  Just
> trying to get this all straight.
>
> Thanks!
>
> stevie
>
> On 12/11/07, hemant kumar <gethemant at gmail.com> wrote:
> > Hi
> >
> > On Tue, 2007-12-11 at 10:53 +0100, Emil Marceta wrote:
> >
> > > Does this means that slave/daemons are not the dependency
anymore?
> >
> > Yes, its gone. bdrb no longer depends on slave and daemons.
> >
> > > By ''not encouraged'' do you mean that 1.0 is not
supporting multiple
> > > threads in the worker or just as a general guidance?
> > >
> > > Could you please comment, how would you approach the following
> > > scenario with 1.0. Currently, we have a worker that creates
threads
> > > that process financial payment transactions. An http request
sends
> > > several 10s or 100s payment transaction records. They are handled
by
> > > the single worker instance. Within the worker there is a pool of
> > > threads created that is calculated based on the number of
> > > transactions. For example for 200 transactions there will be 20
> > > threads where each thread handles 10 requests in a squence. Each
> > > transaction takes about 3-5 seconds, so our throughput is
> > > significantly improved by internal worker parallelization with a
> > > thread pool. The worker periodically updates custom backgroundjob
> > > databse record, so that following ajax request from the client
can
> > > read the status of the worker process. The job is identified with
the
> > > worker key that is stored in the session.
> >
> > Its not encouraged, thats all. You can still have threads in your
> > workers. However, I am planning to add thread pool feature in bdrb
> > itself, that should simplify things a bit.
> >
> > Also ideally, when using EventDriven network programming, you want all
> > your sockets within select loop for efficiency. So, you
wouldn''t need
> > any damn threads, if you can use a HTTP handler that works in Evented
> > manner. What i mean to say is, you don''t do this:
> >
> > a = Net::HTTP.get("http://www.google.com")
> >
> > but you do,
> >
> > Backgroundrb::HTTP.open("http://www.google.com") do |data|
> > process_data(data)
> > end
> >
> > What I am trying to illustrate is, when you ask to open, google.com
> > page, evented model allows you to attach callback ( the block in this
> > case ), which will be called when data arrives from google.com, rather
> > than waiting for it in a thread. So, BackgrounDRb::HTTP.open() returns
> > immediately. And you are concurrent as hell.
> >
> > But this is not possible, because if you are charging cards, then you
> > are probably using ActiveMerchant which is using Net::HTTP and which
> > blocks when you make request. But trust me, writing a simple http
client
> > is not that difficult, there is already connect() available in all
> > workers.
> >
> >
> > > How this works with fastcgi or multiple mongrel based engines
where it
> > > is not guaranteed to hit the same process with the next request?
We
> > > are using custom database tables and code for sharing the status
> > > information now but I was wandering whether the plumbing includes
> > > something to address this.
> >
> > Thats no problem at all, BackgrounDRb is a TCP server, so if you have
> > followed the README file, no matter from which machine, you are making
> > the request if you are specifying worker X, then its guaranteed to hit
> > the same worker(with optional job_key if you are starting your worker
> > dynamically)
> >
> >
> >
> > _______________________________________________
> > Backgroundrb-devel mailing list
> > Backgroundrb-devel at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/backgroundrb-devel
> >
>
>


-- 
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org