Eventmachine talk - Dec 2008 - How to slow down received data rate?

Hi Group,
I''m working on application which acts as a proxy between two servers
and I
have got into following problem:
There is a high speed 100MB link from data source to my proxy and a slow
link from proxy to target server. This is fine for small amounts of data,
but for bigger ones each time EM invokes data_recieved callback on
''source''
connection I have to store received data somewhere while it waits to be
transferred further. This buffer grows very quickly, and this is not
acceptable (what''s more, there is a high delay after source transfers
last
byte and gets response that everything was successfully transferred to
target) :(

+Is there a way in EM for certain connection to slow incoming data rate, or
pause it?+

I guess this would only require leaving ''paused'' sockets alone
when looking
for readable ones.... If it''s not available in current release, does
anybody
know if it will ever be implemented?

Best regards,
Zbigniew Zemla
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/eventmachine-talk/attachments/20081211/287cb84c/attachment.html>

EM does have access to the output buffer size [getOutbound something],
so one way would be to monitor that and have your app send feedback
tothe incoming stream telling it to back off for a bit.
Either that or save all the incoming stuff to disk and write it out
from that as you can.
-=R

On Thu, Dec 11, 2008 at 6:59 AM, Zbigniew Zem?a <z.zemla at gmail.com>
wrote:
> Hi Group,
> I''m working on application which acts as a proxy between two
servers and I
> have got into following problem:
> There is a high speed 100MB link from data source to my proxy and a slow
> link from proxy to target server. This is fine for small amounts of data,
> but for bigger ones each time EM invokes data_recieved callback on
''source''
> connection I have to store received data somewhere while it waits to be
> transferred further. This buffer grows very quickly, and this is not
> acceptable (what''s more, there is a high delay after source
transfers last
> byte and gets response that everything was successfully transferred to
> target) :(
>
> +Is there a way in EM for certain connection to slow incoming data rate, or
> pause it?+
>
> I guess this would only require leaving ''paused'' sockets
alone when looking
> for readable ones.... If it''s not available in current release,
does anybody
> know if it will ever be implemented?
>
> Best regards,
> Zbigniew Zemla
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>


--
Thanks!
-=R

On 11 Dec 2008, at 13:59, Zbigniew Zem?a wrote:
> Hi Group,
Welcome...
> I''m working on application which acts as a proxy between two
servers
> and I have got into following problem:
> There is a high speed 100MB link from data source to my proxy and a
> slow link from proxy to target server. This is fine for small
> amounts of data, but for bigger ones each time EM invokes
> data_recieved callback on ''source'' connection I have to
store
> received data somewhere while it waits to be transferred further.
As Roger suggested, you could use a persisting and out of memory
buffer, however, if the buffer is growing that fast then it sounds
like the kind of situation where you''d just fill a disk, too.
> This buffer grows very quickly, and this is not acceptable (what''s
> more, there is a high delay after source transfers last byte and
> gets response that everything was successfully transferred to
> target) :(
If you''re sending ACKs then why not wait for an ACK before sending a X-
bytes sized packet? You can play with that segment size to optimise /
reduce overhead of the flow control.
> +Is there a way in EM for certain connection to slow incoming data
> rate, or pause it?+
Not natively afaik, we read from the sockets and ACK all the time.
This is intentional, however I can see why it is not ideal for you
right now.
> I guess this would only require leaving ''paused'' sockets
alone when
> looking for readable ones.... If it''s not available in current
> release, does anybody know if it will ever be implemented?
It could be added to the road map, however, using TCP flow
restrictions for this kind of flow control I can''t really recommend.
Is it possible for you to add intelligent flow control to the proxy
application/protocol?
>
>
> Best regards,
> Zbigniew Zemla
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk

Thanks for quick reply.
Sorry, I could have been more precise...
> Either that or save all the incoming stuff to disk and write it out
> from that as you can.
I cannot buffer large amounts of data neither in memory nor on
persistent storage. Even more problematic thing is long interval
between time when client sends last bit and gets success response.
> Is it possible for you to add intelligent flow control to the proxy
> application/protocol?
I cannot implement flow control between source and proxy at
application layer, as this connection is using WebDAV protocol (and I
have no control on client implementation).
What I was thinking about is some solution at transport layer. The
only one I can think of is postponing actual read() from socket at
application layer until enough space in buffer is available, and
letting OS manage this situation (at transport layer). (btw. are there
any problems with this approach / some other better way?)
I know it''s very unusual situation, but I was hoping that maybe EM
somehow supports this.
But I guess not, as James wrote:
> Not natively afaik, we read from the sockets and ACK all the time.
So I guess I''ll need to write some patch implementing the feature I
need... Can someone briefly point me into which classes/methods I
should look into in order to do so?

The solution you''re proposing is deferring read() calls so that data
from
the client will fill the socket kernel buffer, which will cause the kernel
to close the advertised window to slow the sender down.
I think this is pretty broken. First, you''re still buffering large
chunks of
files in memory --- only now you lose some control over how much you buffer,
and you''re guaranteed that it''s buffering instead of draining
out to files
as fast as you can write. Second, you''re messing with TCP congestion
signals
to solve a minor application problem.

Use EventMachine''s popen support to open a pipe to another program that
writes user data to files. That program will block (because the filesystem
blocks), but when it does, the pipe buffer will fill, and
EventMachine''s
#send_data method will transparentl queue the data. Your queued data will
drain as fast as the system can drain it, which is what you want.

Put some benchmarking code in the writer to figure out how many k/sec
you''re
sustaining to the filesystem, and then tune that. Then scale horizontally.

Can you remove descriptors from the read set to implement "deferred
reads"?
I''d be surprised and disappointed to hear that EventMachine supported
that
feature.

On Thu, Dec 11, 2008 at 1:52 PM, Zbigniew Zem?a <z.zemla at gmail.com>
wrote:
> Thanks for quick reply.
> Sorry, I could have been more precise...
>
> > Either that or save all the incoming stuff to disk and write it out
> > from that as you can.
> I cannot buffer large amounts of data neither in memory nor on
> persistent storage. Even more problematic thing is long interval
> between time when client sends last bit and gets success response.
>
> > Is it possible for you to add intelligent flow control to the proxy
> > application/protocol?
> I cannot implement flow control between source and proxy at
> application layer, as this connection is using WebDAV protocol (and I
> have no control on client implementation).
> What I was thinking about is some solution at transport layer. The
> only one I can think of is postponing actual read() from socket at
> application layer until enough space in buffer is available, and
> letting OS manage this situation (at transport layer). (btw. are there
> any problems with this approach / some other better way?)
> I know it''s very unusual situation, but I was hoping that maybe EM
> somehow supports this.
> But I guess not, as James wrote:
> > Not natively afaik, we read from the sockets and ACK all the time.
> So I guess I''ll need to write some patch implementing the feature
I
> need... Can someone briefly point me into which classes/methods I
> should look into in order to do so?
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>


--
---
Thomas H. Ptacek // matasano security
read us on the web: http://www.matasano.com/log
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/eventmachine-talk/attachments/20081211/a0d556d7/attachment-0001.html>

2008/12/11 Thomas Ptacek <tqbf at matasano.com>:
> The solution you''re proposing is deferring read() calls so that
data from
> the client will fill the socket kernel buffer, which will cause the kernel
> to close the advertised window to slow the sender down.
> I think this is pretty broken. First, you''re still buffering large
chunks of
> files in memory --- only now you lose some control over how much you
buffer,
> and you''re guaranteed that it''s buffering instead of
draining out to files
> as fast as you can write. Second, you''re messing with TCP
congestion signals
> to solve a minor application problem.
> Use EventMachine''s popen support to open a pipe to another program
that
> writes user data to files. That program will block (because the filesystem
> blocks), but when it does, the pipe buffer will fill, and
EventMachine''s
> #send_data method will transparentl queue the data. Your queued data will
> drain as fast as the system can drain it, which is what you want.
> Put some benchmarking code in the writer to figure out how many k/sec
you''re
> sustaining to the filesystem, and then tune that. Then scale horizontally.
> Can you remove descriptors from the read set to implement "deferred
reads"?
> I''d be surprised and disappointed to hear that EventMachine
supported that
> feature.
If you really want to implement deferred reads, you can do so by
EM.attach()ing your own socket, and then calling read() (or
nonblocking equivalent) on it only when you are ready to do so:

module DeferredReader
  def initialize sock
    @sock = sock
  end

  def notify_readable
    if ready_to_read?
      receive_data @sock.read(16384)
    end
  rescue EOFError
    detach
  end
end

EM.run{
  sock = TCPSocket.new(''host'', 1234)
  EM.attach sock, DeferredReader, sock
}

  Aman
> On Thu, Dec 11, 2008 at 1:52 PM, Zbigniew Zem?a <z.zemla at
gmail.com> wrote:
>>
>> Thanks for quick reply.
>> Sorry, I could have been more precise...
>>
>> > Either that or save all the incoming stuff to disk and write it
out
>> > from that as you can.
>> I cannot buffer large amounts of data neither in memory nor on
>> persistent storage. Even more problematic thing is long interval
>> between time when client sends last bit and gets success response.
>>
>> > Is it possible for you to add intelligent flow control to the
proxy
>> > application/protocol?
>> I cannot implement flow control between source and proxy at
>> application layer, as this connection is using WebDAV protocol (and I
>> have no control on client implementation).
>> What I was thinking about is some solution at transport layer. The
>> only one I can think of is postponing actual read() from socket at
>> application layer until enough space in buffer is available, and
>> letting OS manage this situation (at transport layer). (btw. are there
>> any problems with this approach / some other better way?)
>> I know it''s very unusual situation, but I was hoping that
maybe EM
>> somehow supports this.
>> But I guess not, as James wrote:
>> > Not natively afaik, we read from the sockets and ACK all the time.
>> So I guess I''ll need to write some patch implementing the
feature I
>> need... Can someone briefly point me into which classes/methods I
>> should look into in order to do so?
>> _______________________________________________
>> Eventmachine-talk mailing list
>> Eventmachine-talk at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>
>
>
> --
> ---
> Thomas H. Ptacek // matasano security
> read us on the web: http://www.matasano.com/log
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>

From: Thomas Ptacek
>
> The solution you''re proposing is deferring read() calls so that
data from the client
> will fill the socket kernel buffer, which will cause the kernel to close
the advertised
> window to slow the sender down.
>
> I think this is pretty broken. First, you''re still buffering large
chunks of files in memory
> --- only now you lose some control over how much you buffer, and
you''re guaranteed
> that it''s buffering instead of draining out to files as fast as
you can write. Second,
> you''re messing with TCP congestion signals to solve a minor
application problem.
Really?  If my application is acting as a proxy, an I have a bunch of untrusted
clients
flooding me with data faster than I can pass it on, then refusing to read the
socket
and letting the kernel buffers fill and letting TCP handle the flow control with
the
remote end seems like _precisely_ the correct solution.

Indeed, any time I''ve contemplated writing an application in
EventMachine (proxy or
not) where untrusted clients are flooding me with data, I have not been able to
work
out any reasonable solution besides deferring reads.  Buffering a boundless
amount
of incoming data temporarily on the hard drive is absolutely not a reasonable
solution from my point of view.

Am I missing something?


Regards,

Bill

Good call--that should probably work [just detach when you''re not
ready to read, then reattach when you are].

2008/12/11 Aman Gupta <themastermind1 at
gmail.com>:
> 2008/12/11 Thomas Ptacek <tqbf at matasano.com>:
>> The solution you''re proposing is deferring read() calls so
that data from
>> the client will fill the socket kernel buffer, which will cause the
kernel
>> to close the advertised window to slow the sender down.
>> I think this is pretty broken. First, you''re still buffering
large chunks of
>> files in memory --- only now you lose some control over how much you
buffer,
>> and you''re guaranteed that it''s buffering instead of
draining out to files
>> as fast as you can write. Second, you''re messing with TCP
congestion signals
>> to solve a minor application problem.
>> Use EventMachine''s popen support to open a pipe to another
program that
>> writes user data to files. That program will block (because the
filesystem
>> blocks), but when it does, the pipe buffer will fill, and
EventMachine''s
>> #send_data method will transparentl queue the data. Your queued data
will
>> drain as fast as the system can drain it, which is what you want.
>> Put some benchmarking code in the writer to figure out how many k/sec
you''re
>> sustaining to the filesystem, and then tune that. Then scale
horizontally.
>> Can you remove descriptors from the read set to implement
"deferred reads"?
>> I''d be surprised and disappointed to hear that EventMachine
supported that
>> feature.
>
> If you really want to implement deferred reads, you can do so by
> EM.attach()ing your own socket, and then calling read() (or
> nonblocking equivalent) on it only when you are ready to do so:
>
> module DeferredReader
>  def initialize sock
>    @sock = sock
>  end
>
>  def notify_readable
>    if ready_to_read?
>      receive_data @sock.read(16384)
>    end
>  rescue EOFError
>    detach
>  end
> end
>
> EM.run{
>  sock = TCPSocket.new(''host'', 1234)
>  EM.attach sock, DeferredReader, sock
> }
>
>  Aman
>
>> On Thu, Dec 11, 2008 at 1:52 PM, Zbigniew Zem?a <z.zemla at
gmail.com> wrote:
>>>
>>> Thanks for quick reply.
>>> Sorry, I could have been more precise...
>>>
>>> > Either that or save all the incoming stuff to disk and write
it out
>>> > from that as you can.
>>> I cannot buffer large amounts of data neither in memory nor on
>>> persistent storage. Even more problematic thing is long interval
>>> between time when client sends last bit and gets success response.
>>>
>>> > Is it possible for you to add intelligent flow control to the
proxy
>>> > application/protocol?
>>> I cannot implement flow control between source and proxy at
>>> application layer, as this connection is using WebDAV protocol (and
I
>>> have no control on client implementation).
>>> What I was thinking about is some solution at transport layer. The
>>> only one I can think of is postponing actual read() from socket at
>>> application layer until enough space in buffer is available, and
>>> letting OS manage this situation (at transport layer). (btw. are
there
>>> any problems with this approach / some other better way?)
>>> I know it''s very unusual situation, but I was hoping that
maybe EM
>>> somehow supports this.
>>> But I guess not, as James wrote:
>>> > Not natively afaik, we read from the sockets and ACK all the
time.
>>> So I guess I''ll need to write some patch implementing the
feature I
>>> need... Can someone briefly point me into which classes/methods I
>>> should look into in order to do so?
>>> _______________________________________________
>>> Eventmachine-talk mailing list
>>> Eventmachine-talk at rubyforge.org
>>> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>>
>>
>>
>> --
>> ---
>> Thomas H. Ptacek // matasano security
>> read us on the web: http://www.matasano.com/log
>>
>> _______________________________________________
>> Eventmachine-talk mailing list
>> Eventmachine-talk at rubyforge.org
>> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>


--
Thanks!
-=R

Hi,

I sent a question about this to the list about eight hours ago, guess it
got stalled in transit... (....a victim of flow control? ;)

From: "James Tucker" <jftucker at
gmail.com>
>
> It could be added to the road map, however, using TCP flow
> restrictions for this kind of flow control I can''t really
recommend.
For an application where untrusted clients are sending data
faster than the data can be processed, using TCP flow control
seems like an ideal solution to me.  I don''t understand why it is
being not-recommended?


Regards,

Bill

Yes, I think you''re missing the fact that you''re trying to
solve a denial of
service attack using TCP flow control. If you''re worried that
you''re being
flooded with data, you''re still worried even after you''ve
closed the
advertised window --- they''ll just add connections.

On Thu, Dec 11, 2008 at 2:39 PM, Bill Kelly <billk at cts.com> wrote:
>
> From: Thomas Ptacek
>
>>
>> The solution you''re proposing is deferring read() calls so
that data from
>> the client
>> will fill the socket kernel buffer, which will cause the kernel to
close
>> the advertised
>> window to slow the sender down.
>>
>> I think this is pretty broken. First, you''re still buffering
large chunks
>> of files in memory
>> --- only now you lose some control over how much you buffer, and
you''re
>> guaranteed
>> that it''s buffering instead of draining out to files as fast
as you can
>> write. Second,
>> you''re messing with TCP congestion signals to solve a minor
application
>> problem.
>>
>
> Really?  If my application is acting as a proxy, an I have a bunch of
> untrusted clients
> flooding me with data faster than I can pass it on, then refusing to read
> the socket
> and letting the kernel buffers fill and letting TCP handle the flow control
> with the
> remote end seems like _precisely_ the correct solution.
>
> Indeed, any time I''ve contemplated writing an application in
EventMachine
> (proxy or
> not) where untrusted clients are flooding me with data, I have not been
> able to work
> out any reasonable solution besides deferring reads.  Buffering a boundless
> amount
> of incoming data temporarily on the hard drive is absolutely not a
> reasonable solution from my point of view.
>
> Am I missing something?
>
>
> Regards,
>
> Bill
>
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>


--
---
Thomas H. Ptacek // matasano security
read us on the web: http://www.matasano.com/log
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/eventmachine-talk/attachments/20081211/9e847916/attachment.html>

On Thu, Dec 11, 2008 at 1:39 PM, Bill Kelly <billk at cts.com> wrote:
> Really?  If my application is acting as a proxy, an I have a bunch of
> untrusted clients
> flooding me with data faster than I can pass it on, then refusing to read
> the socket
> and letting the kernel buffers fill and letting TCP handle the flow control
> with the
> remote end seems like _precisely_ the correct solution.
>
Yes, this is precisely the correct solution.  Also: the clients need not be
malicious.  Perhaps they''re naively giving you data faster than you can
process it.  But really it''s not that naive... this is how TCP is
designed
to operate.  Your program should read data from the socket as fast as you
can process it, and TCP''s flow control will prevent the remote side
from
sending you data until you''re ready.

Indeed, any time I''ve contemplated writing an application in
EventMachine
> (proxy or
> not) where untrusted clients are flooding me with data, I have not been
> able to work
> out any reasonable solution besides deferring reads.  Buffering a boundless
> amount
> of incoming data temporarily on the hard drive is absolutely not a
> reasonable solution from my point of view.
>
> Am I missing something?
>
I don''t think you are... I think this is a rather fundamental problem
with
EventMachine that I don''t know how to solve.

To give some potential solutions: Rev has a "disable" method which
will
suspend event monitoring for any watcher/connection.  This means that the
file descriptor associated with the watcher/connection will no longer be
polled at the kernel until the watcher/connection is enabled again.  So TCP
will cause the remote side to block until you''re ready to process the
data.

With EventMachine, I don''t know of any way to temporarily stop the
event
loop from performing the I/O for an arbitrary connection and calling
receive_data.  I think you''re kind of screwed... you need to buffer the
data
as it''s received and you can''t use normal TCP flow control to
instruct the
remote side not to send data.

Perhaps EventMachine could use something ala Rev''s enable/disable API.

--
Tony Arcieri
medioh.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/eventmachine-talk/attachments/20081211/fdf6a5a8/attachment-0001.html>

From: Thomas Ptacek
>
> Yes, I think you''re missing the fact that you''re trying
to solve
> a denial of service attack using TCP flow control. If you''re
> worried that you''re being flooded with data, you''re still
worried
> even after you''ve closed the advertised window ---
they''ll just
> add connections.
Ah.  I was viewing it more as a continuum.  Certainly at the top
end there are the massive DDoS attacks that nobody can defend
against--my provider will likely suspend my services for the
duration of the attack.  And then the sort of DoS attack you
describe above, for which, when one has determined such a thing
is occurring, I imagine it would be preferable to just blackhole
the source IP at the router or kernel level.  But I had in mind,
to use Tony''s terminology, more of a naive client than a
malicious client.

I suspect I should re-think my previous statement about
applications vs. proxies.  With applications built on custom
protocols, one can certainly design flow control into the
protocol, and simply drop clients who don''t follow the protocol.

With proxies, though, where one is dealing with pre-existing
protocols, I think the question of naive clients reenters the
picture.

In that case it still seems perfectly reasonable to me to let
TCP handle the flow control when a naive client is sending data
too fast.  Unless we are convinced it is always better to
buffer that data locally on the HDD.  I''m not able to convince
myself buffering will always be preferable.  (?)


Regards,

Bill

I honestly think this is a design problem on the app level.

On 12 Dec 2008, at 06:31, Bill Kelly wrote:
>
> From: Thomas Ptacek
>>
>> Yes, I think you''re missing the fact that you''re
trying to solve
>> a denial of service attack using TCP flow control. If you''re
>> worried that you''re being flooded with data, you''re
still worried
>> even after you''ve closed the advertised window ---
they''ll just
>> add connections.
>
> Ah.  I was viewing it more as a continuum.  Certainly at the top
> end there are the massive DDoS attacks that nobody can defend
> against--my provider will likely suspend my services for the
> duration of the attack.  And then the sort of DoS attack you
> describe above, for which, when one has determined such a thing
> is occurring, I imagine it would be preferable to just blackhole
> the source IP at the router or kernel level.  But I had in mind,
> to use Tony''s terminology, more of a naive client than a
> malicious client.
You see, in my opinion, I would say that if the clients are typically
that naive, the protocol is not being used in a suitable manner, or
has been badly selected.

If you''re pushing data at a rate that is way over CPU bound,
you''re
always going to have these issues. If you know your problem is CPU
bound, why not use a chunked request protocol?
> I suspect I should re-think my previous statement about
> applications vs. proxies.  With applications built on custom
> protocols, one can certainly design flow control into the
> protocol, and simply drop clients who don''t follow the protocol.
Precisely, and I''d generally recommend doing that asap.
> With proxies, though, where one is dealing with pre-existing
> protocols, I think the question of naive clients reenters the
> picture.
I can see how large amounts of asynchrony in between the channels is a
problem for proxies, especially when it''s consistent as described. The
problem should really be driven by the writer, rather than the reader,
that is, read when the write buffer is getting low. This is very
similar to how the fast file send stuff works, the difference being
only that the read is from disk.

The problem with using TCP flow control, is that at times, you can
bottleneck and cause timeouts or overfill buffers. That becomes a real
pain.
> In that case it still seems perfectly reasonable to me to let
> TCP handle the flow control when a naive client is sending data
> too fast.  Unless we are convinced it is always better to buffer
> that data locally on the HDD.  I''m not able to convince
> myself buffering will always be preferable.  (?)
Buffering will certainly not always be preferable, however, either
you''re heading for real overflow (kinda DoS by design) or not. The
boundaries of which are really where I start to feel there is design
smell. Buffering web requests for 10-50ms on a high load proxy is not
going to be a major issue on it''s own. What the description of the OPs
problem seems to be however, is that there is a large incoming stream
being proxied out via a really small pipe. The issue with this design
is clear - you''re always running on a backlog. If you buffer, you OOME
or run out of space, if you don''t buffer, you run on massive choke.
Often that kind of choking can be a problem for upstream servers too,
indeed that''s often one of the most common reasons to have a caching
proxy. I''d still recommend a flow control protocol for handling this,
even though it may be more work.
> Regards,
>
> Bill
>
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk

On 11 Dec 2008, at 19:52, Zbigniew Zem?a wrote:
> I cannot implement flow control between source and proxy at
> application layer, as this connection is using WebDAV protocol (and I
> have no control on client implementation).
Have you considered using Squid?

James Tucker:
> Have you considered using Squid?
Yes, I''m already using it - but it only helps when it comes to
accelerating download of data from target site. Heh.. another thing
which I haven''t thought about is that I''ll be filling up
internal
socket between squid and application if I delay reading from socket...
I wonder how squid will react...

>From Tony:
> With EventMachine, I don''t know of any way to temporarily stop the
event
> loop from performing the I/O for an arbitrary connection and calling
> receive_data.  I think you''re kind of screwed... you need to
buffer the data
> as it''s received and you can''t use normal TCP flow
control to instruct the
> remote side not to send data.
I''m exactly in this situation
> Perhaps EventMachine could use something ala Rev''s enable/disable
API.
would be great if it did so (although I know it''s not a commonly used
feature)



2008/12/12 Tony Arcieri <tony at medioh.com>:
> On Thu, Dec 11, 2008 at 1:39 PM, Bill Kelly <billk at cts.com> wrote:
>>
>> Really?  If my application is acting as a proxy, an I have a bunch of
>> untrusted clients
>> flooding me with data faster than I can pass it on, then refusing to
read
>> the socket
>> and letting the kernel buffers fill and letting TCP handle the flow
>> control with the
>> remote end seems like _precisely_ the correct solution.
>
> Yes, this is precisely the correct solution.  Also: the clients need not be
> malicious.  Perhaps they''re naively giving you data faster than
you can
> process it.  But really it''s not that naive... this is how TCP is
designed
> to operate.  Your program should read data from the socket as fast as you
> can process it, and TCP''s flow control will prevent the remote
side from
> sending you data until you''re ready.
>
>> Indeed, any time I''ve contemplated writing an application in
EventMachine
>> (proxy or
>> not) where untrusted clients are flooding me with data, I have not been
>> able to work
>> out any reasonable solution besides deferring reads.  Buffering a
>> boundless amount
>> of incoming data temporarily on the hard drive is absolutely not a
>> reasonable solution from my point of view.
>>
>> Am I missing something?
>
> I don''t think you are... I think this is a rather fundamental
problem with
> EventMachine that I don''t know how to solve.
>
> To give some potential solutions: Rev has a "disable" method
which will
> suspend event monitoring for any watcher/connection.  This means that the
> file descriptor associated with the watcher/connection will no longer be
> polled at the kernel until the watcher/connection is enabled again.  So TCP
> will cause the remote side to block until you''re ready to process
the data.
>
> With EventMachine, I don''t know of any way to temporarily stop the
event
> loop from performing the I/O for an arbitrary connection and calling
> receive_data.  I think you''re kind of screwed... you need to
buffer the data
> as it''s received and you can''t use normal TCP flow
control to instruct the
> remote side not to send data.
>
> Perhaps EventMachine could use something ala Rev''s enable/disable
API.
>
> --
> Tony Arcieri
> medioh.com
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>

On Fri, Dec 12, 2008 at 5:13 AM, James Tucker <jftucker at gmail.com>
wrote:
> You see, in my opinion, I would say that if the clients are typically that
> naive, the protocol is not being used in a suitable manner, or has been
> badly selected.
>
Virtually every other asynchronous system I''ve worked with gives you
the
ability to take advantage of TCP flow control.  One of the foremost that
comes to mind is Erlang, which gives you the option to have incoming TCP
data delivered over the its actor protocol.  The messages are buffered in a
given process''s mailbox.  This would have the unfortunate problem of
messages being delivered faster than they are being consumed, leading to
unchecked growth in the size of the incoming mailbox.  Erlang solved this by
letting you specify that you only want one message delivered to a given
process''s mailbox.  When a socket becomes readable, Erlang will perform
the
I/O and deliver the data over the actor protocol.  It will then stop polling
that socket.  Once you''ve processed that message, you can specify you
want
the next message delivered.

And as I mentioned earlier, Rev gives you a method you can call to prevent
the connection from being polled at the kernel level.  When you''re
ready to
consume data again, you flip polling back on.  Why can''t EventMachine
do
that?

why not use a chunked request protocol?
>
While in certain cases that''s applicable, if you''re suggesting
that as a
general solution I really think you''re trying to force transport layer
concerns up into the application layer, which is silly.

--
Tony Arcieri
medioh.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://rubyforge.org/pipermail/eventmachine-talk/attachments/20081212/15906288/attachment.html>

> And as I mentioned earlier, Rev gives you a method you can call to prevent
> the connection from being polled at the kernel level. ?When you''re
ready to
> consume data again, you flip polling back on. ?Why can''t
EventMachine do
> that?
Aman described how to defer reads that way in EM.
It is possible to detach an active connection using
EM::Connection#detach
(which returns the file descriptor) you can keep that around and later
you can
attach it to the event loop using EM.attach

oldmoe
espace.com.eg
oldmoe.blogspot.com

> Aman described how to defer reads that way in EM.
> It is possible to detach an active connection using
> EM::Connection#detach
> (which returns the file descriptor) you can keep that around and later
> you can
> attach it to the event loop using EM.attach
Note that EM.attach/detach do not play well with EM''s read buffer, so
if you''re using them you must provide a notify_readable callback and
handle reading from the socket yourself. Calling detach on a regular
EM descriptor (one that uses receive_data) will lead to unexpected
results as data might still be sitting in EM''s read and write buffers.

  Aman
>
> oldmoe
> espace.com.eg
> oldmoe.blogspot.com
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>

Wouldn''t the read buffers be flushed when in a recieve_data call?

On Dec 13, 3:52?am, "Aman Gupta" <themastermi... at gmail.com>
wrote:
> > Aman described how to defer reads that way in EM.
> > It is possible to detach an active connection using
> > EM::Connection#detach
> > (which returns the file descriptor) you can keep that around and later
> > you can
> > attach it to the event loop using EM.attach
>
> Note that EM.attach/detach do not play well with EM''s read buffer,
so
> if you''re using them you must provide a notify_readable callback
and
> handle reading from the socket yourself. Calling detach on a regular
> EM descriptor (one that uses receive_data) will lead to unexpected
> results as data might still be sitting in EM''s read and write
buffers.
>
> ? Aman
>
>
>
> > oldmoe
> > espace.com.eg
> > oldmoe.blogspot.com
> > _______________________________________________
> > Eventmachine-talk mailing list
> > Eventmachine-t... at rubyforge.org
> >http://rubyforge.org/mailman/listinfo/eventmachine-talk
>
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-t... at
rubyforge.orghttp://rubyforge.org/mailman/listinfo/eventmachine-talk