Eventmachine talk - Jul 2008 - local source port/address bind()ing for

From: "Eric Monti" <emonti at
matasano.com>
>
> In short, ''bind'' only really seems to get exposed by
EventMachine for
> UDP datagram sockets. I need to do this sort of thing with TCP to
> speak a particular protocol through someone''s paranoid firewall.
It''s
> also an option I''d really like to support in some generalized
network
> testing tools I''m writing around evma.
We may need to submit patches.

In a somewhat similar situation (discussed on this list back in April),
it turns out to be possible to have EM create a TCP server on an
ephemeral port--but no way to determine what that port number is.

Which is something we''d need if we were to implement an FTP client
initiating a data transfer in "active" mode, or an FTP server in
"passive" mode.

(One side creates a TCP listen socket on an ephemeral port, then
transmits that port number across the command channel to the other
end, so the other end knows what data port to connect to.)


Regards,

Bill

On 16 Jul 2008, at 08:02, Bill Kelly wrote:
>
> From: "Eric Monti" <emonti at matasano.com>
>>
>> In short, ''bind'' only really seems to get exposed by
EventMachine
>> for  UDP datagram sockets. I need to do this sort of thing with TCP
>> to  speak a particular protocol through someone''s paranoid
>> firewall. It''s  also an option I''d really like to
support in some
>> generalized network  testing tools I''m writing around evma.
>
> We may need to submit patches.
>
> In a somewhat similar situation (discussed on this list back in
> April),
> it turns out to be possible to have EM create a TCP server on an
> ephemeral port--but no way to determine what that port number is.
iirc that patch went in.
> Which is something we''d need if we were to implement an FTP client
> initiating a data transfer in "active" mode, or an FTP server in
> "passive" mode.
ruftpd is coming :)
> (One side creates a TCP listen socket on an ephemeral port, then
> transmits that port number across the command channel to the other
> end, so the other end knows what data port to connect to.)
>
>
> Regards,
>
> Bill
>
>
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk

On Jul 16, 2008, at 7:47 AM, James Tucker wrote:
>
> On 16 Jul 2008, at 08:02, Bill Kelly wrote:
>
>>
>> From: "Eric Monti" <emonti at matasano.com>
>>>
>>> In short, ''bind'' only really seems to get exposed
by EventMachine
>>> for  UDP datagram sockets. I need to do this sort of thing with
>>> TCP to  speak a particular protocol through someone''s
paranoid
>>> firewall. It''s  also an option I''d really like to
support in some
>>> generalized network  testing tools I''m writing around
evma.
>>
>> We may need to submit patches.
>>
>> In a somewhat similar situation (discussed on this list back in
>> April),
>> it turns out to be possible to have EM create a TCP server on an
>> ephemeral port--but no way to determine what that port number is.
>
> iirc that patch went in.
Indeed it did. get_sockname() has been exposed since 0.12.0. Another
feature I "really needed" :)

So, I looked at the feasibility of a patch to the extension for this
while I was "digging around". Haven''t really come up with an
elegant/
appropriate way to do it yet. I''m wary of hacking out a patch with
only a minimal understanding of some of the design issues.

The C/C++ extension has ephemeral TCP ports for clients baked in to
existing code. A patch for this will be less trivial than the one for
get_sockname().

Eric Monti
Matasano Security
emonti at matasano.com


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2421 bytes
Desc: not available
URL:
<http://rubyforge.org/pipermail/eventmachine-talk/attachments/20080716/257d3cdb/attachment.bin>

>>> From: "Eric Monti" <emonti at matasano.com>
>>>>
>>>> In short, ''bind'' only really seems to get
exposed by EventMachine for
>>>>  UDP datagram sockets. I need to do this sort of thing with TCP
to  speak a
>>>> particular protocol through someone''s paranoid
firewall. It''s  also an
>>>> option I''d really like to support in some generalized
network  testing tools
>>>> I''m writing around evma.
>>>
>>> We may need to submit patches.
>>>
>>> In a somewhat similar situation (discussed on this list back in
April),
>>> it turns out to be possible to have EM create a TCP server on an
>>> ephemeral port--but no way to determine what that port number is.
>>
>> iirc that patch went in.
>
> Indeed it did. get_sockname() has been exposed since 0.12.0. Another
feature
> I "really needed" :)
>
> So, I looked at the feasibility of a patch to the extension for this while
I
> was "digging around". Haven''t really come up with an
elegant/appropriate way
> to do it yet. I''m wary of hacking out a patch with only a minimal
> understanding of some of the design issues.
>
Yeah you can bind to ephemeral ports using the latest SVN version [see
http://rubyeventmachine.com/wiki/CodeSnippets last entry].
If I understand correctly the OP wanted to bind an outgoing TCP Port
to a certain port number.  That would probably require some hacking in
the C, but would make a useful addition.  I believe rev exposes raw
sockets, I''m not sure if the pure ruby version of EM also does.

-R

On 16 Jul 2008, at 16:26, Roger Pack wrote:
>>>> From: "Eric Monti" <emonti at matasano.com>
>>>>>
>>>>> In short, ''bind'' only really seems to get
exposed by
>>>>> EventMachine for
>>>>> UDP datagram sockets. I need to do this sort of thing with
TCP
>>>>> to  speak a
>>>>> particular protocol through someone''s paranoid
firewall. It''s
>>>>> also an
>>>>> option I''d really like to support in some
generalized network
>>>>> testing tools
>>>>> I''m writing around evma.
>>>>
>>>> We may need to submit patches.
>>>>
>>>> In a somewhat similar situation (discussed on this list back in
>>>> April),
>>>> it turns out to be possible to have EM create a TCP server on
an
>>>> ephemeral port--but no way to determine what that port number
is.
>>>
>>> iirc that patch went in.
>>
>> Indeed it did. get_sockname() has been exposed since 0.12.0.
>> Another feature
>> I "really needed" :)
>>
>> So, I looked at the feasibility of a patch to the extension for
>> this while I
>> was "digging around". Haven''t really come up with an
elegant/
>> appropriate way
>> to do it yet. I''m wary of hacking out a patch with only a
minimal
>> understanding of some of the design issues.
A research patch can even be helpful sometimes.
> Yeah you can bind to ephemeral ports using the latest SVN version [see
> http://rubyeventmachine.com/wiki/CodeSnippets last entry].
> If I understand correctly the OP wanted to bind an outgoing TCP Port
> to a certain port number.  That would probably require some hacking in
> the C, but would make a useful addition.  I believe rev exposes raw
> sockets, I''m not sure if the pure ruby version of EM also does.
iirc it''s possible to get at the fd. Obviously this doesn''t
help if
it''s already had to be established through some api.

What could be really useful (under a number of recent topics) would be
to have the ability to supply established sockets (or in fact anything
selectable), to the reactor core. This could solve one of the problems
under the postgresql async api thread, whereby what is really required
is to know when select returns a value. Under that API, one would want
the ability to prevent EventMachine from consuming the input (from
calling read), as you ideally want to call ConsumeInput instead.

I went some way toward this with a very naive patch, however it does
not disable the read currently, and I believe has other problems.
>
>
> -R
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk

On Jul 16, 12:30 pm, James Tucker <jftuc... at gmail.com> wrote:
 >
> What could be really useful (under a number of recent topics) would be
> to have the ability to supply established sockets (or in fact anything
> selectable), to the reactor core. This could solve one of the problems
> under the postgresql async api thread, whereby what is really required
> is to know when select returns a value. Under that API, one would want
> the ability to prevent EventMachine from consuming the input (from
> calling read), as you ideally want to call ConsumeInput instead.
>
> I went some way toward this with a very naive patch, however it does
> not disable the read currently, and I believe has other problems.
>


I''m not too clear on how this would work in the context of pgsql. In
my naivete I''d assumed the best way to do an evented pgsql client
would be to just implement its wire protocol.

If we were to integrate arbitrary fds into the EM core, the following
requrements would apply:
1) You''d have to allow the socket to be set nonblocking and CLOEXEC
(the latter is Unix-only, of course),
2) You''d have to allow EM to do all subsequent reads and writes on it.
3) For Ruby integration, there''d need to be a wrapper object
associated with the fd to receive the events. (Possibly a subclass of
EM::Connection.

Given that, it should be fairly easy.

On Jul 16, 2008, at 10:26 AM, Roger Pack wrote:
> Yeah you can bind to ephemeral ports using the latest SVN version [see
> http://rubyeventmachine.com/wiki/CodeSnippets last entry].
> If I understand correctly the OP wanted to bind an outgoing TCP Port
> to a certain port number.  That would probably require some hacking
> in the
> C, but would make a useful addition.
>
To be clear: Not just source port. Source address too. Multi-homed
hosts and weird network routing setups may require this in many cases.
Also, I misspoke about "bind()" only exposed for datagram sockets. I
meant that from the "client" perspective only. It is ofcourse exposed
(though not directly) for UDP *and* TCP servers currently as well.

Just got swamped with work, so I may not get another stab for a while.
But, if nobody else does first, I''ll try my hand at a patch when I
next get some free time.
>
> I believe rev exposes raw
> sockets, I''m not sure if the pure ruby version of EM also does.
"raw sockets" as in "SOCK_RAW"? That''s definitely
interesting. Hadn''t
expected to see that feature for a while but it definitely has
exciting possibilities.

Eric Monti
emonti at matasano.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2421 bytes
Desc: not available
URL:
<http://rubyforge.org/pipermail/eventmachine-talk/attachments/20080716/55a126c6/attachment.bin>

>> I believe rev exposes raw
>> sockets, I''m not sure if the pure ruby version of EM also
does.
>
> "raw sockets" as in "SOCK_RAW"? That''s
definitely interesting. Hadn''t
> expected to see that feature for a while but it definitely has exciting
> possibilities.
I was just referring to the fd itself :)
A google search seems to reveal that ruby has something todo with
SOCK_RAW, as well, though I''ve never used it.  That could indeed be
fun to play with :)

With regard to Francis''
> 2) You''d have to allow EM to do all subsequent reads and writes on
it.
I believe the postgres adapter prefers to do its own reads and writes
on sockets [and handles the IO in C].  It just needs to be notified
when the socket is readable.  So it would require some modification to
EM''s core for this to happen.  Or to postgres.  The former being that
which would make EM more flexible.
My $0.02
-R