Howdy, I have a server application (Unix-based) which, in addition to accepting telnet connections from clients, also popen''s several processes from which it receives events locally. I''ve been wanting to convert this application from ruby threads to EventMachine, but realized I didn''t know how to mesh the popen''s with EM. Is there a way? Regards, Bill
On 4/21/07, Bill Kelly <billk at cts.com> wrote:> Howdy, > > I have a server application (Unix-based) which, in addition > to accepting telnet connections from clients, also popen''s > several processes from which it receives events locally. > > I''ve been wanting to convert this application from ruby > threads to EventMachine, but realized I didn''t know how to > mesh the popen''s with EM. > > Is there a way? >Hmm, yes - easiest is perhaps, using slave library(written by Ara - search ruby-talk, rubyforge), to fork processes. slave library hands you a nice Lifeline to manage children and hence you can receive message from child and write to child and stuff like that. If you want to to use popen, then i believe with EM it will block your main eventloop, until you are done with pipe. Francis, can shed more light on this, but this should be trivial to verify. slave uses Drb to manage childs. -- gnufied
From: "hemant" <gethemant at gmail.com>> > On 4/21/07, Bill Kelly <billk at cts.com> wrote: >> Howdy, >> >> I have a server application (Unix-based) which, in addition >> to accepting telnet connections from clients, also popen''s >> several processes from which it receives events locally. >> >> I''ve been wanting to convert this application from ruby >> threads to EventMachine, but realized I didn''t know how to >> mesh the popen''s with EM. >> >> Is there a way? >> > > Hmm, yes - easiest is perhaps, using slave library(written by Ara - > search ruby-talk, rubyforge), to fork processes. slave library hands > you a nice Lifeline to manage children and hence you can receive > message from child and write to child and stuff like that. > > If you want to to use popen, then i believe with EM it will block your > main eventloop, until you are done with pipe. Francis, can shed more > light on this, but this should be trivial to verify.Windows OS aside (which I don''t care about for this app), the file descriptors returned by popen should work with select(), and should support NONBLOCK semantics on Unix. ((I only mention Windows because I realize pipes are a problem on that OS.)) So I was thinking, if only I could just "hand off" these file descriptors returned by popen to EventMachine, they should work fine with EM''s select-based architecture. Regards, Bill
On 4/21/07, Bill Kelly <billk at cts.com> wrote:> > > Hmm, yes - easiest is perhaps, using slave library(written by Ara - > > search ruby-talk, rubyforge), to fork processes. slave library hands > > you a nice Lifeline to manage children and hence you can receive > > message from child and write to child and stuff like that. > > > > If you want to to use popen, then i believe with EM it will block your > > main eventloop, until you are done with pipe. Francis, can shed more > > light on this, but this should be trivial to verify. > > Windows OS aside (which I don''t care about for this app), the > file descriptors returned by popen should work with select(), > and should support NONBLOCK semantics on Unix. ((I only mention > Windows because I realize pipes are a problem on that OS.)) > > So I was thinking, if only I could just "hand off" these > file descriptors returned by popen to EventMachine, they should > work fine with EM''s select-based architecture.YES. This is something I''ve wanted to do for a long time, and Twisted definitely has it. (Twisted also has an asynchronous fork, which is a simpler version of the same idea.) In fact, I already started working on this once. If you look in the EM source, you''ll see that there is some support for nonblocking file i/o there already, just not exposed to the APIs yet. When I get a few minutes, I''ll pull it back out and see what kind of shape it''s in. Bill and Hemant, can you guys suggest a potential app-level API that would be good for something like this? popen is usually used with a block, correct? Maybe this: EventMachine.popen {....}, {...} EventMachine.fork {...}, {...} where the second block is executed in case of an error? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070421/ed763a76/attachment-0001.html
From: Francis Cianfrocca> > Bill and Hemant, can you guys suggest a potential app-level > API that would be good for something like this? popen is usually > used with a block, correct? > > Maybe this: > > EventMachine.popen {....}, {...} > EventMachine.fork {...}, {...} > > where the second block is executed in case of an error?The block-forms of fork and popen are nice, but in my current situation with popen (and my most recent use of fork), I couldn''t use the block semantics. With popen, my situation is this: I start an external program which monitors a remote server, and returns a stream of status information. I want the pipe to this external program to remain open indefinitely. (It crashes sometimes, in which case I restart it.) Essentially, I just want to get receive_data events in a "normal EventMachine way" from the pipe returned by popen, as long as the external process remains alive. To me, it''s "just another streaming file descriptor" from which I would like to get receive_data events. Regarding fork, my most recent use was rather similar to my popen case above. I had tens of thousands of IP addresses I wanted to do ''gethostbyname'' on, and it was taking forever to do them serially. So I created pipes and pre-forked 50 workers. The parent kept the pipes in a freelist, grabbed pipes from the freelist and fed them a single IP, and pushed the pipe onto a busy list. The parent would select on the pipes in the busy list, read the results from any who were ready, and move them back on the freelist. Nothing too fancy, but worked great. But in both of these cases, the block-form of popen and fork don''t seem appropriate.>From the parent''s point of view, I''d just like to seepipes as "just another eventable file descriptor". (From the forked child''s point of view, running in a block/proc seems just fine.) Does this make sense? Regards, Bill
On 4/22/07, Bill Kelly <billk at cts.com> wrote:> > From: Francis Cianfrocca > > > > Bill and Hemant, can you guys suggest a potential app-level > > API that would be good for something like this? popen is usually > > used with a block, correct? > > > > Maybe this: > > > > EventMachine.popen {....}, {...} > > EventMachine.fork {...}, {...} > > > > where the second block is executed in case of an error? > > The block-forms of fork and popen are nice, but in my > current situation with popen (and my most recent use of > fork), I couldn''t use the block semantics. > > With popen, my situation is this: I start an external > program which monitors a remote server, and returns a > stream of status information. I want the pipe to this > external program to remain open indefinitely. (It > crashes sometimes, in which case I restart it.) > > Essentially, I just want to get receive_data events > in a "normal EventMachine way" from the pipe returned > by popen, as long as the external process remains > alive. > > To me, it''s "just another streaming file descriptor" > from which I would like to get receive_data events. > > > Regarding fork, my most recent use was rather similar > to my popen case above. I had tens of thousands of > IP addresses I wanted to do ''gethostbyname'' on, and > it was taking forever to do them serially. So I > created pipes and pre-forked 50 workers. > > The parent kept the pipes in a freelist, grabbed pipes > from the freelist and fed them a single IP, and pushed > the pipe onto a busy list. > > The parent would select on the pipes in the busy list, > read the results from any who were ready, and move them > back on the freelist. > > Nothing too fancy, but worked great. > > > But in both of these cases, the block-form of popen > and fork don''t seem appropriate. > > >From the parent''s point of view, I''d just like to see > pipes as "just another eventable file descriptor". >I think this would require too many changes in EventMachine core.But idea is definitely good. What I would like is, fork child and ability to pass messages to the child asyhcronously and set callback handlers from parent, which would be invoked by child. For example: # In parent child_object = EventMachine.fork( class/module which would run as child) connect(child_object, SIGNAL(foo),self, foo_handler) so basically, when child emits signal foo then foo_handler would be called in parent. I know I am asking for too much. Taking Bill''s example for running gethostbyname() on tens of thousand of machines, above example would work quite nicely. But may be running just a gethostbyname() is not a perfect use case of above thing. But for forking workers, which are CPU bound, above approach would be awesome. You get erlang like message passing suddenly in Ruby, which can scale to any number of cores.!!> (From the forked child''s point of view, running in a > block/proc seems just fine.) > > > Does this make sense? > > > Regards, > > Bill > > > > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >-- gnufied ----------- There was only one Road; that it was like a great river: its springs were at every doorstep, and every path was its tributary. http://people.inxsasia.com/hemant
On 4/22/07, hemant <gethemant at gmail.com> wrote:> > What I would like is, fork child and ability to pass messages to the > child asyhcronously and set callback handlers from parent, which would > be invoked by child. > > For example: > > # In parent > child_object = EventMachine.fork( class/module which would run as child) > connect(child_object, SIGNAL(foo),self, foo_handler) > > > so basically, when child emits signal foo then foo_handler would be > called in parent. I know I am asking for too much.By adding in the ability to send messages asynchronously to a fork child, you''re taking this to another level altogether, and in fact the general case would simply any set of (unrelated) processes messaging each other in patterns of arbitrary complexity. You can model this in at least three different ways that I know of (distributed objects like dRb, message-queueing, and publish-subscribe), and I definitely think EM should support this kind of thing. In fact, I think EM should be the *platform of choice* for applications like this. To deal with the fork/popen issue more narrowly for the moment: we could do something like Twisted, and treat the fork child as something that fires a completion event. In other words, EM is basically just wrapping the SIGCHLD and the waitpid handling. (It''s quite a bit more work on Windows of course, but the principle is the same.) Doing something like CGI-style streaming through file descriptors is also a good idea. But to the more general case, where unrelated processes are messaging each other: I think we should really look at making a Ruby DSL that would define the coordination and the messaging patterns among the different processes. Erlang does this at the language level of course, but we can do it at the library level. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070422/3b32c540/attachment.html
From: "hemant" <gethemant at gmail.com>> > On 4/22/07, Bill Kelly <billk at cts.com> wrote: >> >> From the parent''s point of view, I''d just like to see >> pipes as "just another eventable file descriptor". > > I think this would require too many changes in EventMachine core.But > idea is definitely good. > > What I would like is, fork child and ability to pass messages to the > child asyhcronously and set callback handlers from parent, which would > be invoked by child. > > For example: > > # In parent > child_object = EventMachine.fork( class/module which would run as child) > connect(child_object, SIGNAL(foo),self, foo_handler) > > > so basically, when child emits signal foo then foo_handler would be > called in parent. I know I am asking for too much.I guess I''m confused, because what you''ve written above looks almost like what I thought I was asking for. :) Except I''m just interested in the pipes, not so much the signals. It would be fine with me, for example, if EM just by default converted SIGPIPE and SIGCHLD into CONNECTION_UNBOUND events, I would think. (EM would handle reaping the zombie for me.) But let me put fork aside for the moment and just look at the popen case. I don''t understand why it would require any changes to EM''s core to treat the pipe returned by popen as ''just another eventable file descriptor''? Looking at em.cpp, we already have the beginnings of event support on nonblocking file descriptors with _OpenFileForWriting. Why couldn''t the pipe returned by popen be wrapped in a FileStreamDescriptor in the same manner? Rather than requiring changes to EM''s core, it seems like it would fit in quite naturally. Am I missing something? Regards, Bill
On 4/22/07, Bill Kelly <billk at cts.com> wrote:> > From: "hemant" <gethemant at gmail.com> > > > > On 4/22/07, Bill Kelly <billk at cts.com> wrote: > >> > >> From the parent''s point of view, I''d just like to see > >> pipes as "just another eventable file descriptor". > > > > I think this would require too many changes in EventMachine core.But > > idea is definitely good. > > > > What I would like is, fork child and ability to pass messages to the > > child asyhcronously and set callback handlers from parent, which would > > be invoked by child. > > > > For example: > > > > # In parent > > child_object = EventMachine.fork( class/module which would run as child) > > connect(child_object, SIGNAL(foo),self, foo_handler) > > > > > > so basically, when child emits signal foo then foo_handler would be > > called in parent. I know I am asking for too much. > > I guess I''m confused, because what you''ve written above looks > almost like what I thought I was asking for. :) Except I''m > just interested in the pipes, not so much the signals. It > would be fine with me, for example, if EM just by default > converted SIGPIPE and SIGCHLD into CONNECTION_UNBOUND events, > I would think. (EM would handle reaping the zombie for me.) > > But let me put fork aside for the moment and just look at the > popen case. > > I don''t understand why it would require any changes to EM''s > core to treat the pipe returned by popen as ''just another > eventable file descriptor''? Looking at em.cpp, we already > have the beginnings of event support on nonblocking file > descriptors with _OpenFileForWriting. Why couldn''t the pipe > returned by popen be wrapped in a FileStreamDescriptor in > the same manner? > > Rather than requiring changes to EM''s core, it seems like it > would fit in quite naturally. Am I missing something? > >Bill, I haven''t seen C++ internals of EM. Francis, I typed a long mail with some ideas and then cut saved it elsewhere, because i thought, I should back them up with some code.
On 4/23/07, hemant <gethemant at gmail.com> wrote:> On 4/22/07, Bill Kelly <billk at cts.com> wrote: > > > > From: "hemant" <gethemant at gmail.com> > > > > > > On 4/22/07, Bill Kelly <billk at cts.com> wrote: > > >> > > >> From the parent''s point of view, I''d just like to see > > >> pipes as "just another eventable file descriptor". > > > > > > I think this would require too many changes in EventMachine core.But > > > idea is definitely good. > > > > > > What I would like is, fork child and ability to pass messages to the > > > child asyhcronously and set callback handlers from parent, which would > > > be invoked by child. > > > > > > For example: > > > > > > # In parent > > > child_object = EventMachine.fork( class/module which would run as child) > > > connect(child_object, SIGNAL(foo),self, foo_handler) > > > > > > > > > so basically, when child emits signal foo then foo_handler would be > > > called in parent. I know I am asking for too much. > > > > I guess I''m confused, because what you''ve written above looks > > almost like what I thought I was asking for. :) Except I''m > > just interested in the pipes, not so much the signals. It > > would be fine with me, for example, if EM just by default > > converted SIGPIPE and SIGCHLD into CONNECTION_UNBOUND events, > > I would think. (EM would handle reaping the zombie for me.) > > > > But let me put fork aside for the moment and just look at the > > popen case. > > > > I don''t understand why it would require any changes to EM''s > > core to treat the pipe returned by popen as ''just another > > eventable file descriptor''? Looking at em.cpp, we already > > have the beginnings of event support on nonblocking file > > descriptors with _OpenFileForWriting. Why couldn''t the pipe > > returned by popen be wrapped in a FileStreamDescriptor in > > the same manner? > > > > Rather than requiring changes to EM''s core, it seems like it > > would fit in quite naturally. Am I missing something? > > > > > > Bill, I haven''t seen C++ internals of EM. Francis, I typed a long mail > with some ideas and then cut saved it elsewhere, because i thought, I > should back them up with some code. >BTW, whats the current development trunk of EM on rubyforge svn. SVN trunk looks a bit messed up, with trunk containing just a README file.
From: "hemant" <gethemant at gmail.com>> > BTW, whats the current development trunk of EM on rubyforge svn. SVN > trunk looks a bit messed up, with trunk containing just a README file.I asked Francis about that just yesterday in an email. :) Francis'' reply: version_0 is the active branch. The reason for the version_1 branch is that Jeff Rose at one time was doing some active dev there on a pure-Ruby version with a somewhat different architecture but that line isn''t going to pan out. The existing C++ EM in version_0 will continue to be supported indefinitely, even if someday there are alternative implementations with different APIs. Regards, Bill
On 4/22/07, Bill Kelly <billk at cts.com> wrote:> > > I guess I''m confused, because what you''ve written above looks > almost like what I thought I was asking for. :) Except I''m > just interested in the pipes, not so much the signals. It > would be fine with me, for example, if EM just by default > converted SIGPIPE and SIGCHLD into CONNECTION_UNBOUND events, > I would think. (EM would handle reaping the zombie for me.) > > But let me put fork aside for the moment and just look at the > popen case. > > I don''t understand why it would require any changes to EM''s > core to treat the pipe returned by popen as ''just another > eventable file descriptor''? Looking at em.cpp, we already > have the beginnings of event support on nonblocking file > descriptors with _OpenFileForWriting. Why couldn''t the pipe > returned by popen be wrapped in a FileStreamDescriptor in > the same manner? > > Rather than requiring changes to EM''s core, it seems like it > would fit in quite naturally. Am I missing something?You''re probably right. I''m going to try to get to this today. To add a bit to the point about signals: EM should completely hide them (and the parallel mechanism on Windows) from user code. I think with Twisted you get an event (which is triggered by a signal under the covers) that tells you the fork child has ended, which is all you really care about, of course. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070423/f6042af6/attachment.html