thr3ads.net - Eventmachine talk - [Eventmachine-talk] How to keep a pipe open [Nov 2007]

If this information is useful, please help other people find it:
Share via:

Duane Johnson

2007-Nov-27 14:49 UTC

[Eventmachine-talk] How to keep a pipe open

Hi all,

I''ve been reading what I can from the list regarding
EventMachine::popen.  The following two messages were helpful (for
anyone else trying to use it):

http://rubyforge.org/pipermail/eventmachine-talk/2007-June/000747.html
http://rubyforge.org/pipermail/eventmachine-talk/2007-November/
001227.html

My question is this: How do I keep a pipe open for interactive-shell-
type commands that "hang up"?

I''m not sure I totally understand the mechanisms in place that are
causing this, so please feel free to chime in with obvious answers
here.  As an example of the problem, take "TeX":
> EventMachine::run do
>   EventMachine::popen("tex -ipc", TexInstance)
> end
This results in the following output (TexInstance simply forwards
what it receives):

received: This is TeX, Version 3.141592 (Web2C 7.5.6)
received: **
received: ! End of file on the terminal... why?

This is the exact same message I receive if I start tex up and hit
"ctrl-d" on the keyboard... so by its appearance, it seems like my
pipe is closing because tex is receiving an end-of-file character.

Any ideas or workarounds?  Thanks!

Duane Johnson
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20071127/053ca112/attachment.html

Roger Pack

2007-Nov-27 19:02 UTC

head link

[Eventmachine-talk] How to keep a pipe open

Maybe try it with a process where you can see what it receives more
easily?  Maybe the process is being interrupted...is your ruby program
terminating?
Take care.

On Nov 27, 2007 3:49 PM, Duane Johnson <canadaduane at gmail.com>
wrote:> Hi all,
>
> I''ve been reading what I can from the list regarding
EventMachine::popen.
> The following two messages were helpful (for anyone else trying to use it):
>
> http://rubyforge.org/pipermail/eventmachine-talk/2007-June/000747.html
> http://rubyforge.org/pipermail/eventmachine-talk/2007-November/001227.html
>
> My question is this: How do I keep a pipe open for interactive-shell-type
> commands that "hang up"?
>
> I''m not sure I totally understand the mechanisms in place that are
causing
> this, so please feel free to chime in with obvious answers here.  As an
> example of the problem, take "TeX":
>
>
> EventMachine::run do
>   EventMachine::popen("tex -ipc", TexInstance)
> end
> This results in the following output (TexInstance simply forwards what it
> receives):
>
> received: This is TeX, Version 3.141592 (Web2C 7.5.6)
> received: **
> received: ! End of file on the terminal... why?
>
> This is the exact same message I receive if I start tex up and hit
"ctrl-d"
> on the keyboard... so by its appearance, it seems like my pipe is closing
> because tex is receiving an end-of-file character.
>
> Any ideas or workarounds?  Thanks!
>
> Duane Johnson
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>


--
-Roger Pack
For God hath not given us the spirit of fear; but of power, and of
love, and of a sound mind" -- 2 Timothy 1:7

Duane Johnson

2007-Nov-27 23:58 UTC

head link

[Eventmachine-talk] How to keep a pipe open

What kind of process would you suggest?  I tried "telnet" and the
result was different: there was no "hang up"... but then again, ctrl-
d sent to telnet on the shell does nothing either.

The ruby program does not terminate in the case of popen("tex",
TexInstance).  The pipe closes but the ruby program remains running
in the EM run loop.

Duane

On Nov 27, 2007, at 8:02 PM, Roger Pack wrote:
> Maybe try it with a process where you can see what it receives more
> easily?  Maybe the process is being interrupted...is your ruby program
> terminating?
> Take care.
>
> On Nov 27, 2007 3:49 PM, Duane Johnson <canadaduane at gmail.com>
wrote:
>> Hi all,
>>
>> I''ve been reading what I can from the list regarding
>> EventMachine::popen.
>> The following two messages were helpful (for anyone else trying to
>> use it):
>>
>> http://rubyforge.org/pipermail/eventmachine-talk/2007-June/
>> 000747.html
>> http://rubyforge.org/pipermail/eventmachine-talk/2007-November/
>> 001227.html
>>
>> My question is this: How do I keep a pipe open for interactive-
>> shell-type
>> commands that "hang up"?
>>
>> I''m not sure I totally understand the mechanisms in place that
are
>> causing
>> this, so please feel free to chime in with obvious answers here.
>> As an
>> example of the problem, take "TeX":
>>
>>
>> EventMachine::run do
>>   EventMachine::popen("tex -ipc", TexInstance)
>> end
>> This results in the following output (TexInstance simply forwards
>> what it
>> receives):
>>
>> received: This is TeX, Version 3.141592 (Web2C 7.5.6)
>> received: **
>> received: ! End of file on the terminal... why?
>>
>> This is the exact same message I receive if I start tex up and hit
>> "ctrl-d"
>> on the keyboard... so by its appearance, it seems like my pipe is
>> closing
>> because tex is receiving an end-of-file character.
>>
>> Any ideas or workarounds?  Thanks!
>>
>> Duane Johnson
>> _______________________________________________
>> Eventmachine-talk mailing list
>> Eventmachine-talk at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>>
>
>
>
> --
> -Roger Pack
> For God hath not given us the spirit of fear; but of power, and of
> love, and of a sound mind" -- 2 Timothy 1:7
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk

Duane Johnson

2007-Nov-28 14:56 UTC

head link

[Eventmachine-talk] How to keep a pipe open

Here''s some more information.  The following is a complete listing of
my test app:
> require ''rubygems''
> require ''eventmachine''
> require ''fileutils''
>
> class TexInstance < EventMachine::Connection
>   def receive_data(data)
>     print data
>   end
> end
>
> class GlueServer < EventMachine::Connection
>   def initialize(data)
>     puts "New connection: #{@client_id} #{data.inspect}"
>     super(data)
>   end
>
>   def receive_data(command)
>     # command = command.chomp
>     print "send: #{command}"
>     $instance.send_data command
>   end
> end
>
>
> EventMachine::run do
>   $instance = EventMachine::popen("tex", TexInstance)
>   EventMachine::start_server "127.0.0.1", 4100, GlueServer
> end

Before I can telnet in to port 4100 to have commands forwarded to the
TexInstance / tex command, I receive the EOF error message (in TeX''s
peculiar terminology, "! End of file on the terminal... why?").

Now take a look at this:
> IO.popen("ftp", "r+") do |io|
>   while true do
>     puts io.read(1)
>     #io.puts "test"
>   end
> end
The above simple ruby program behaves as I expect it to: TeX does NOT
receive an EOF from the ruby program, and just waits for input.
What''s going on here?  What''s the difference between
ruby''s popen and
EM''s popen?

Thanks,
Duane Johnson

Duane Johnson

2007-Nov-29 13:51 UTC

head link

[Eventmachine-talk] How to keep a pipe open

Ah-hah!

I think I''ve found the problem.  Inside the Socketpair constructor on
line 1386 of em.cpp, it sets both the input AND output sockets to non-
blocking:
> 	// Set the socketpair nonblocking. Obviously DON''T set CLOEXEC.
> 	if (!SetSocketNonblocking (sv[0]) || !SetSocketNonblocking (sv[1])) {
> 		close (sv[0]);
> 		close (sv[1]);
> 		return NULL;
> 	}

By removing the SetSocketNonblocking(sv[1]) from the line, the TeX
command-line application pauses and waits for EventMachine to send
data, as expected.

If anyone is more familiar with this subject, I would appreciate some
feedback here.  Is there some reason the output stream is set to non-
blocking?

Thanks,
Duane Johnson

Roger Pack

2007-Nov-29 18:19 UTC

head link

[Eventmachine-talk] How to keep a pipe open

maybe there''s an extra close in there?

On Nov 29, 2007 2:51 PM, Duane Johnson <canadaduane at gmail.com>
wrote:> Ah-hah!
>
> I think I''ve found the problem.  Inside the Socketpair constructor
on
> line 1386 of em.cpp, it sets both the input AND output sockets to non-
> blocking:
>
> >       // Set the socketpair nonblocking. Obviously DON''T set
CLOEXEC.
> >       if (!SetSocketNonblocking (sv[0]) || !SetSocketNonblocking
(sv[1])) {
> >               close (sv[0]);
> >               close (sv[1]);
> >               return NULL;
> >       }
>
>
> By removing the SetSocketNonblocking(sv[1]) from the line, the TeX
> command-line application pauses and waits for EventMachine to send
> data, as expected.
>
> If anyone is more familiar with this subject, I would appreciate some
> feedback here.  Is there some reason the output stream is set to non-
> blocking?
>
>
> Thanks,
> Duane Johnson
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>


--
-Roger Pack
For God hath not given us the spirit of fear; but of power, and of
love, and of a sound mind" -- 2 Timothy 1:7

Duane Johnson

2007-Nov-30 13:41 UTC

head link

[Eventmachine-talk] How to keep a pipe open

Thanks for your reply, Roger.  No, I think it''s necessary to close
both sockets--they are to be closed when there is an error condition
(that''s what the if statement is for--it checks if there was an error
while setting the socket nonblocking).

So basically, this is the new, correct code:

	// Set the socketpair nonblocking. Obviously DON''T set CLOEXEC.
	if (!SetSocketNonblocking (sv[0])) {
		close (sv[0]);
		close (sv[1]);
		return NULL;
	}

Can this change go into the core?

Duane


On Nov 29, 2007, at 7:19 PM, Roger Pack wrote:
> maybe there''s an extra close in there?
>
> On Nov 29, 2007 2:51 PM, Duane Johnson <canadaduane at gmail.com>
wrote:
>> Ah-hah!
>>
>> I think I''ve found the problem.  Inside the Socketpair
constructor on
>> line 1386 of em.cpp, it sets both the input AND output sockets to
>> non-
>> blocking:
>>
>>>       // Set the socketpair nonblocking. Obviously DON''T
set
>>> CLOEXEC.
>>>       if (!SetSocketNonblocking (sv[0]) || !SetSocketNonblocking
>>> (sv[1])) {
>>>               close (sv[0]);
>>>               close (sv[1]);
>>>               return NULL;
>>>       }
>>
>>
>> By removing the SetSocketNonblocking(sv[1]) from the line, the TeX
>> command-line application pauses and waits for EventMachine to send
>> data, as expected.
>>
>> If anyone is more familiar with this subject, I would appreciate some
>> feedback here.  Is there some reason the output stream is set to non-
>> blocking?
>>
>>
>> Thanks,
>> Duane Johnson
>> _______________________________________________
>> Eventmachine-talk mailing list
>> Eventmachine-talk at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>>
>
>
>
> --
> -Roger Pack
> For God hath not given us the spirit of fear; but of power, and of
> love, and of a sound mind" -- 2 Timothy 1:7
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk

Francis Cianfrocca

2007-Nov-30 15:16 UTC

head link

[Eventmachine-talk] How to keep a pipe open

On Nov 30, 2007 4:41 PM, Duane Johnson <canadaduane at gmail.com> wrote:
> Thanks for your reply, Roger.  No, I think it''s necessary to close
> both sockets--they are to be closed when there is an error condition
> (that''s what the if statement is for--it checks if there was an
error
> while setting the socket nonblocking).
>
> So basically, this is the new, correct code:
>
>        // Set the socketpair nonblocking. Obviously DON''T set
CLOEXEC.
>        if (!SetSocketNonblocking (sv[0])) {
>                close (sv[0]);
>                close (sv[1]);
>                return NULL;
>        }
>
> Can this change go into the core?
>

I can do that as soon as I''m sure that the sv[1] descriptor is never
read by
the reactor.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20071130/769ed3bc/attachment.html

Bill Kelly

2007-Nov-30 15:44 UTC

head link

[Eventmachine-talk] How to keep a pipe open

From: Francis Cianfrocca>
> On Nov 30, 2007 4:41 PM, Duane Johnson <canadaduane at gmail.com>
wrote:
> >
> >        // Set the socketpair nonblocking. Obviously DON''T set
CLOEXEC.
> >
> >        if (!SetSocketNonblocking (sv[0])) {
> >
> >                close (sv[0]);
> >                close (sv[1]);
> >                return NULL;
> >        }
> >
> > Can this change go into the core?
>
> I can do that as soon as I''m sure that the sv[1] descriptor is
never
> read by the reactor.
If I''m not mistaken, the parent closes sv[1] right after the fork()
anyway.


Regards,

Bill

Francis Cianfrocca

2007-Nov-30 17:52 UTC

head link

[Eventmachine-talk] How to keep a pipe open

On Nov 30, 2007 6:44 PM, Bill Kelly <billk at cts.com> wrote:
>
> If I''m not mistaken, the parent closes sv[1] right after the
fork()
> anyway.
>

You''ve used this feature as much as anyone, Bill. Any reason not to
make
this change?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20071130/b376d3a0/attachment.html

Bill Kelly

2007-Nov-30 18:33 UTC

head link

[Eventmachine-talk] How to keep a pipe open

From: Francis Cianfrocca>
> On Nov 30, 2007 6:44 PM, Bill Kelly <billk at cts.com> wrote:
> >
> > If I''m not mistaken, the parent closes sv[1] right after the
fork()
> > anyway.
>
> You''ve used this feature as much as anyone, Bill. Any reason not
to
> make this change?
It looks like the Right Thing To Do(tm).

EM (the parent) only cares about sv[0], and the child only
cares about sv[1].

And it makes sense that a typical child program would NOT
expect its stdin/stdout to be in non-blocking mode by default.

Regards,

Bill

Bill Kelly

2007-Dec-01 00:55 UTC

head link

[Eventmachine-talk] How to keep a pipe open

From: "Bill Kelly" <billk at cts.com>>
> From: Francis Cianfrocca
>>
>> On Nov 30, 2007 6:44 PM, Bill Kelly <billk at cts.com> wrote:
>> >
>> > If I''m not mistaken, the parent closes sv[1] right after
the fork()
>> > anyway.
>>
>> You''ve used this feature as much as anyone, Bill. Any reason
not to
>> make this change?
>
> It looks like the Right Thing To Do(tm).
>
> EM (the parent) only cares about sv[0], and the child only
> cares about sv[1].
>
> And it makes sense that a typical child program would NOT
> expect its stdin/stdout to be in non-blocking mode by default.
BTW, I''m not sure how that came across on re-reading it... but
it was (meant) in the style of me thinking aloud, rather than
some sort of scolding pontification. <grin>


Regards,

Bill

Francis Cianfrocca

2007-Dec-01 05:51 UTC

head link

[Eventmachine-talk] How to keep a pipe open

On Dec 1, 2007 3:55 AM, Bill Kelly <billk at cts.com> wrote:
>
> From: "Bill Kelly" <billk at cts.com>
> >
> > From: Francis Cianfrocca
> >>
> >> On Nov 30, 2007 6:44 PM, Bill Kelly <billk at cts.com>
wrote:
> >> >
> >> > If I''m not mistaken, the parent closes sv[1] right
after the fork()
> >> > anyway.
> >>
> >> You''ve used this feature as much as anyone, Bill. Any
reason not to
> >> make this change?
> >
> > It looks like the Right Thing To Do(tm).
> >
> > EM (the parent) only cares about sv[0], and the child only
> > cares about sv[1].
> >
> > And it makes sense that a typical child program would NOT
> > expect its stdin/stdout to be in non-blocking mode by default.
>
> BTW, I''m not sure how that came across on re-reading it... but
> it was (meant) in the style of me thinking aloud, rather than
> some sort of scolding pontification. <grin>
>
>
Changes made and checked in. Thanks for the discussion, all.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20071201/ad15c1b0/attachment-0001.html

Duane Johnson

2007-Dec-01 08:16 UTC

head link

[Eventmachine-talk] How to keep a pipe open

On Dec 1, 2007, at 6:51 AM, Francis Cianfrocca wrote:
> On Dec 1, 2007 3:55 AM, Bill Kelly <billk at cts.com> wrote:
>
> From: "Bill Kelly" <billk at cts.com>
> >
> > From: Francis Cianfrocca
> >>
> >> On Nov 30, 2007 6:44 PM, Bill Kelly < billk at cts.com>
wrote:
> >> >
> >> > If I''m not mistaken, the parent closes sv[1] right
after the
> fork()
> >> > anyway.
> >>
> >> You''ve used this feature as much as anyone, Bill. Any
reason not to
> >> make this change?
> >
> > It looks like the Right Thing To Do(tm).
> >
> > EM (the parent) only cares about sv[0], and the child only
> > cares about sv[1].
> >
> > And it makes sense that a typical child program would NOT
> > expect its stdin/stdout to be in non-blocking mode by default.
>
> BTW, I''m not sure how that came across on re-reading it... but
> it was (meant) in the style of me thinking aloud, rather than
> some sort of scolding pontification. <grin>
>
>
> Changes made and checked in. Thanks for the discussion, all.

Thanks, Francis & Bill.

-- Duane Johnson
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20071201/93ea3284/attachment.html

Michael S. Fischer

2007-Dec-05 15:30 UTC

head link

[Eventmachine-talk] How to keep a pipe open

On a related note, now that the changes to make the input descriptor
blocking on a popen''ed process are in:

How can we then close the input descriptor w/o closing the output one? i.e.,
if you''ve popen''ed a process and want to send_data to it, then
close the
connection yet still receive data until the child process closes its
standard output (i.e., our input), how can we do that?

It would be great if we could do something like this, and receive the
expected result:

require ''eventmachine''

module CatHandler
        def post_init
                send_data "this is a test\n"
                close_connection_after_writing
        end

        def receive_data data
                print data
        end

        def unbind
                EventMachine::stop_event_loop
        end
end

EventMachine::run {
        EventMachine::popen("cat", CatHandler)
}

Likewise, I''ve run into some issues because EM::popen only lets the
caller
access the standard output stream of the subprocess and no others (in
particular, standard error).  Today I compensate for all of these
deficiencies by doing something kludgey like:

EventMachine::popen(%[sh -c "echo #{data} | #{command} #{args.join}
2>&1"])

Observation:  Because they both involve Connection objects, a popen is
treated the same as a network connection, despite the fact that in some very
important ways they are quite different:  popens have different attributes
(pid and exit status vs. peer IPs and ports) and have access to more than
one output stream.   In addition, unlike TCP sockets, the application
designer often may want to access the output that''s provided after the
program''s input stream has been flushed and closed.

Perhaps it''s time to subclass Connection so that we can take advantage
of
the differences while impacting existing socket code as little as possible?

Best regards,

--Michael

On Nov 27, 2007 2:49 PM, Duane Johnson <canadaduane at gmail.com> wrote:
> Hi all,
> I''ve been reading what I can from the list regarding
EventMachine::popen.
> The following two messages were helpful (for anyone else trying to use it):
>
> http://rubyforge.org/pipermail/eventmachine-talk/2007-June/000747.html
> http://rubyforge.org/pipermail/eventmachine-talk/2007-November/001227.html
>
> My question is this: How do I keep a pipe open for interactive-shell-type
> commands that "hang up"?
>
> I''m not sure I totally understand the mechanisms in place that are
causing
> this, so please feel free to chime in with obvious answers here.  As an
> example of the problem, take "TeX":
>
> EventMachine::run do
>   EventMachine::popen("tex -ipc", TexInstance)
> end
>
>
> This results in the following output (TexInstance simply forwards what it
> receives):
>
> received: This is TeX, Version 3.141592 (Web2C 7.5.6)
> received: **
> received: ! End of file on the terminal... why?
>
> This is the exact same message I receive if I start tex up and hit
> "ctrl-d" on the keyboard... so by its appearance, it seems like
my pipe is
> closing because tex is receiving an end-of-file character.
>
> Any ideas or workarounds?  Thanks!
>
> Duane Johnson
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20071205/b305c521/attachment.html

Eventmachine talk - Nov 2007 - How to keep a pipe open

[Eventmachine-talk] How to keep a pipe open

[Eventmachine-talk] How to keep a pipe open

[Eventmachine-talk] How to keep a pipe open

[Eventmachine-talk] How to keep a pipe open

[Eventmachine-talk] How to keep a pipe open

[Eventmachine-talk] How to keep a pipe open

[Eventmachine-talk] How to keep a pipe open

[Eventmachine-talk] How to keep a pipe open

[Eventmachine-talk] How to keep a pipe open

[Eventmachine-talk] How to keep a pipe open

[Eventmachine-talk] How to keep a pipe open

[Eventmachine-talk] How to keep a pipe open

[Eventmachine-talk] How to keep a pipe open

[Eventmachine-talk] How to keep a pipe open

[Eventmachine-talk] How to keep a pipe open