Is there anyway I can force each "send_data" call to send seperately ? Sometimes 2 responses get stuffed together and my client parser doesn''t know what to make of it..
On 12/29/06, Daniel Aquino <mr.danielaquino at gmail.com> wrote:> Is there anyway I can force each "send_data" call to send seperately ? > > Sometimes 2 responses get stuffed together and my client parser doesn''t > know what to make of it.. > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >What do you mean by "send separately"? TCP doesn''t respect message boundaries, everything is just a stream of bytes, and they are allowed to be fragmented arbitrarily (so long as they are not delivered out of order). Have you considered using UDP for your application?
If the data is coming in in a stream how does the server know what the boundaries are for a chunk of data ? Francis Cianfrocca wrote:> On 12/29/06, Daniel Aquino <mr.danielaquino at gmail.com> wrote: >> Is there anyway I can force each "send_data" call to send seperately ? >> >> Sometimes 2 responses get stuffed together and my client parser doesn''t >> know what to make of it.. >> _______________________________________________ >> Eventmachine-talk mailing list >> Eventmachine-talk at rubyforge.org >> http://rubyforge.org/mailman/listinfo/eventmachine-talk >> > > What do you mean by "send separately"? TCP doesn''t respect message > boundaries, everything is just a stream of bytes, and they are allowed > to be fragmented arbitrarily (so long as they are not delivered out of > order). > > Have you considered using UDP for your application? > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >
For something like that you''re in need of a tool for tokenizing the
input
data. I''ve written something to do just that:
gem install buftok
(see also http://rubyforge.org/projects/buftok for the RubyForge page)
In your Module (or subclassed Connection) define the following
def post_init
@buffer = BufferedTokenizer.new(token)
end
Where the token is the boundary you''re delimiting by (if you just do
BufferedTokenizer.new it will default to newline)
Then in your receive_data method:
def receive_data(data)
@buffer.extract(data).map { |message| receive_message message }
end
And finally:
def receive_message(message)
...
end
This will call the receive_message method whenever the delimiting token is
encoutered, giving to you a complete, reassembled buffer containing the
entire message.
--
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
(970) 232-4208
On 12/30/06, Daniel Aquino <mr.danielaquino at gmail.com>
wrote:>
> If the data is coming in in a stream how does the server know what the
> boundaries are for a chunk of data ?
>
> Francis Cianfrocca wrote:
> > On 12/29/06, Daniel Aquino <mr.danielaquino at gmail.com> wrote:
> >> Is there anyway I can force each "send_data" call to
send seperately ?
> >>
> >> Sometimes 2 responses get stuffed together and my client parser
doesn''t
> >> know what to make of it..
> >> _______________________________________________
> >> Eventmachine-talk mailing list
> >> Eventmachine-talk at rubyforge.org
> >> http://rubyforge.org/mailman/listinfo/eventmachine-talk
> >>
> >
> > What do you mean by "send separately"? TCP doesn''t
respect message
> > boundaries, everything is just a stream of bytes, and they are allowed
> > to be fragmented arbitrarily (so long as they are not delivered out of
> > order).
> >
> > Have you considered using UDP for your application?
> > _______________________________________________
> > Eventmachine-talk mailing list
> > Eventmachine-talk at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> >
> _______________________________________________
> Eventmachine-talk mailing list
> Eventmachine-talk at rubyforge.org
> http://rubyforge.org/mailman/listinfo/eventmachine-talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20061230/fff02ce8/attachment-0001.html
Oh, a note on the above:
@buffer.extract(data).each { |message| receive_message message }
would suffice for that implementation. When I typed map I was thinking more
in the context of some intermediate message processing.
For example, if you were doing newline-delimited JSON messages:
begin
require ''fjson''
rescue LoadError
require ''json''
end
def post_init
@buffer = BufferedTokenizer.new
end
def receive_data(data)
@buffer.extract(data).map { |message| JSON.parse message }.each {
|message| receive_message message }
end
def receive_message(message)
...
end
will extract newline-delimited JSON objects
You can also add:
def send_message(message)
send_data message.to_json + ''\n''
end
To encode and send JSON messages for you.
If your messages just consist of hashes and arrays (possibly nested) of
strings and numbers, this is all you really need.
- Tony
On 12/30/06, Tony Arcieri <tony at clickcaster.com>
wrote:>
> For something like that you''re in need of a tool for tokenizing
the input
> data. I''ve written something to do just that:
>
> gem install buftok
>
> (see also http://rubyforge.org/projects/buftok for the RubyForge page)
>
> In your Module (or subclassed Connection) define the following
>
> def post_init
> @buffer = BufferedTokenizer.new(token)
> end
>
> Where the token is the boundary you''re delimiting by (if you just
do
> BufferedTokenizer.new it will default to newline)
>
> Then in your receive_data method:
>
> def receive_data(data)
> @buffer.extract(data).map { |message| receive_message message }
> end
>
> And finally:
>
> def receive_message(message)
> ...
> end
>
> This will call the receive_message method whenever the delimiting token is
> encoutered, giving to you a complete, reassembled buffer containing the
> entire message.
>
> --
> Tony Arcieri
> ClickCaster, Inc.
> tony at clickcaster.com
> (970) 232-4208
>
> On 12/30/06, Daniel Aquino <mr.danielaquino at gmail.com> wrote:
> >
> > If the data is coming in in a stream how does the server know what the
> > boundaries are for a chunk of data ?
> >
> > Francis Cianfrocca wrote:
> > > On 12/29/06, Daniel Aquino < mr.danielaquino at gmail.com>
wrote:
> > >> Is there anyway I can force each "send_data" call
to send seperately
> > ?
> > >>
> > >> Sometimes 2 responses get stuffed together and my client
parser
> > doesn''t
> > >> know what to make of it..
> > >> _______________________________________________
> > >> Eventmachine-talk mailing list
> > >> Eventmachine-talk at rubyforge.org
> > >> http://rubyforge.org/mailman/listinfo/eventmachine-talk
> > >>
> > >
> > > What do you mean by "send separately"? TCP
doesn''t respect message
> > > boundaries, everything is just a stream of bytes, and they are
allowed
> > > to be fragmented arbitrarily (so long as they are not delivered
out of
> > > order).
> > >
> > > Have you considered using UDP for your application?
> > > _______________________________________________
> > > Eventmachine-talk mailing list
> > > Eventmachine-talk at rubyforge.org
> > > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> > >
> > _______________________________________________
> > Eventmachine-talk mailing list
> > Eventmachine-talk at rubyforge.org
> > http://rubyforge.org/mailman/listinfo/eventmachine-talk
> >
>
--
Tony Arcieri
ClickCaster, Inc.
tony at clickcaster.com
(970) 232-4208
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://rubyforge.org/pipermail/eventmachine-talk/attachments/20061230/c7845fdf/attachment.html
On 12/30/06, Tony Arcieri <tony at clickcaster.com> wrote:> > Oh, a note on the above: > > @buffer.extract(data).each { |message| receive_message message } > > would suffice for that implementation. When I typed map I was thinking > more in the context of some intermediate message processing.this looks really cool, Tony. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20061231/ed4a701e/attachment.html
Well, let me know if you''d be interested in integrating it directly into EventMachine somehow (perhaps as a BufferedConnection subclass?) I released it under the Ruby license but I''m down for whatever - Tony On 12/31/06, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> > On 12/30/06, Tony Arcieri <tony at clickcaster.com> wrote: > > > > Oh, a note on the above: > > > > @buffer.extract(data).each { |message| receive_message message } > > > > would suffice for that implementation. When I typed map I was thinking > > more in the context of some intermediate message processing. > > > > > this looks really cool, Tony. > > > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk > >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20061231/98fb0b7c/attachment.html
EM is licensed under either GPL or Ruby license (user''s choice). thanks for the offer, I take a close look and let you know. Did you write any introductory material on how to understand and use this code? On 12/31/06, Tony Arcieri <tony at clickcaster.com> wrote:> > Well, let me know if you''d be interested in integrating it directly into > EventMachine somehow (perhaps as a BufferedConnection subclass?) > > I released it under the Ruby license but I''m down for whatever > > - Tony > > On 12/31/06, Francis Cianfrocca <garbagecat10 at gmail.com> wrote: > > > On 12/30/06, Tony Arcieri <tony at clickcaster.com> wrote: > > > > > > Oh, a note on the above: > > > > > > @buffer.extract(data).each { |message| receive_message message } > > > > > > would suffice for that implementation. When I typed map I was > > > thinking more in the context of some intermediate message processing. > > > > > > > > > > this looks really cool, Tony. > > > > > > _______________________________________________ > > Eventmachine-talk mailing list > > Eventmachine-talk at rubyforge.org > > http://rubyforge.org/mailman/listinfo/eventmachine-talk > > > > > > > -- > Tony Arcieri > ClickCaster, Inc. > tony at clickcaster.com > (970) 232-4208 > > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20061231/dc913719/attachment-0001.html
The source (and comments) are about it in terms of documentation... and the implementation is relatively simple http://pastie.caboo.se/30456 -- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 On 12/31/06, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> > EM is licensed under either GPL or Ruby license (user''s choice). thanks > for the offer, I take a close look and let you know. Did you write any > introductory material on how to understand and use this code? > > On 12/31/06, Tony Arcieri <tony at clickcaster.com> wrote: > > > > Well, let me know if you''d be interested in integrating it directly into > > EventMachine somehow (perhaps as a BufferedConnection subclass?) > > > > I released it under the Ruby license but I''m down for whatever > > > > - Tony > > > > On 12/31/06, Francis Cianfrocca < garbagecat10 at gmail.com> wrote: > > > > > On 12/30/06, Tony Arcieri <tony at clickcaster.com> wrote: > > > > > > > > Oh, a note on the above: > > > > > > > > @buffer.extract(data).each { |message| receive_message message } > > > > > > > > would suffice for that implementation. When I typed map I was > > > > thinking more in the context of some intermediate message processing. > > > > > > > > > > > > > > > this looks really cool, Tony. > > > > > > > > > _______________________________________________ > > > Eventmachine-talk mailing list > > > Eventmachine-talk at rubyforge.org > > > http://rubyforge.org/mailman/listinfo/eventmachine-talk > > > > > > > > > > > > -- > > Tony Arcieri > > ClickCaster, Inc. > > tony at clickcaster.com > > (970) 232-4208 > > > > _______________________________________________ > > Eventmachine-talk mailing list > > Eventmachine-talk at rubyforge.org > > http://rubyforge.org/mailman/listinfo/eventmachine-talk > > > > > > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20061231/b381dade/attachment.html
Because the "chunks" have length fields, or a terminating sequence of bytes. If they don''t, your protocol is broken. On 12/30/06, Daniel Aquino <mr.danielaquino at gmail.com> wrote:> If the data is coming in in a stream how does the server know what the > boundaries are for a chunk of data ?
On 1/1/07, Thomas Ptacek <tqbf at matasano.com> wrote:> > Because the "chunks" have length fields, or a terminating sequence of > bytes. > > If they don''t, your protocol is broken. > > On 12/30/06, Daniel Aquino <mr.danielaquino at gmail.com> wrote: > > If the data is coming in in a stream how does the server know what the > > boundaries are for a chunk of data ? > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >Generally speaking, with TCP there''s no meaningful message-level boundaries in the data stream that you get from EventMachine::Connection. You basically have to buffer up the data you get from #receive_data and be prepared to keep it in a buffer across multiple calls to #receive_data. This would be an example of how to handle data that is organized into lines (not to say that this is the best way. See also EventMachine::Protocols:::LineAndTextProtocol): class MyHandler < EventMachine::Connection def initialize args super @data = "" end def receive_data data @data << data while newline = @data.index("\n") line = @data.slice!(0..newline).chomp process_line( line ) end def process_line line # do something with the line, or have a subclass implement this method end end EventMachine.run { EventMachine.start_server "127.0.0.1", 8910, MyServer } -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070101/026fa37d/attachment.html
You might want to look at the comments of the implementation I pastied for exactly why you shouldn''t do it that way. @data << data That will necessarily scan both strings twice, create a new object (allocating enough memory to hold them both), then perform a copy operation for both. Bottom line: this implementation scales O(n^2) for the size of the buffer. The implementation I gave scales O(n). See: http://pastie.caboo.se/30456 - Tony On 1/1/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> > On 1/1/07, Thomas Ptacek <tqbf at matasano.com> wrote: > > > > Because the "chunks" have length fields, or a terminating sequence of > > bytes. > > > > If they don''t, your protocol is broken. > > > > On 12/30/06, Daniel Aquino <mr.danielaquino at gmail.com > wrote: > > > If the data is coming in in a stream how does the server know what the > > > boundaries are for a chunk of data ? > > _______________________________________________ > > Eventmachine-talk mailing list > > Eventmachine-talk at rubyforge.org > > http://rubyforge.org/mailman/listinfo/eventmachine-talk > > > > > Generally speaking, with TCP there''s no meaningful message-level > boundaries in the data stream that you get from EventMachine::Connection. > You basically have to buffer up the data you get from #receive_data and be > prepared to keep it in a buffer across multiple calls to #receive_data. > This would be an example of how to handle data that is organized into lines > (not to say that this is the best way. See also > EventMachine::Protocols:::LineAndTextProtocol): > > class MyHandler < EventMachine::Connection > def initialize args > super > @data = "" > end > > def receive_data data > @data << data > while newline = @data.index ("\n") > line = @data.slice!(0..newline).chomp > process_line( line ) > end > > def process_line line > # do something with the line, or have a subclass implement this method > end > end > > EventMachine.run { > EventMachine.start_server "127.0.0.1", 8910, MyServer > } > > > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk > >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070101/7ebbb7b4/attachment.html
On 1/1/07, Tony Arcieri <tony at clickcaster.com> wrote:> > You might want to look at the comments of the implementation I pastied for > exactly why you shouldn''t do it that way. > > @data << data > > That will necessarily scan both strings twice, create a new object > (allocating enough memory to hold them both), then perform a copy operation > for both. Bottom line: this implementation scales O(n^2) for the size of > the buffer. > > The implementation I gave scales O(n). See: > > http://pastie.caboo.se/30456Yes, I see the point, you''re using the array to save the multiple allocations and paying a little extra in the scan. Have you profiled it? Would it be easy to adapt this for use with protocols like HTTP and SIP, that alternate newline-delimited lines with data blocks that are delimited either by a predetermined length or by the end of the stream? More practically, can you implement EventMachine::Protocols::LineAndTextProtocol with BufferedTokenizer? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070101/00fec177/attachment-0001.html
So, sorry about the delay on getting back to you on this... Looking at LineAndTextProtocol, I''m wondering why it takes a buffer size. Why not just fall back on the typical pattern for reading binary data, i.e. it''s made available as it''s read. If the desired read buffer is exceeded, you could have a #fill method to dump data back into the tokenizer''s buffer. Replacing the line parser was fairly simple, but I''m confused as to why its API is designed in that way. Does the HTTP code request the entire Content-Length (which is then buffered), or just small chunks at a time? - Tony On 1/1/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> > On 1/1/07, Tony Arcieri <tony at clickcaster.com> wrote: > > > > You might want to look at the comments of the implementation I pastied > > for exactly why you shouldn''t do it that way. > > > > @data << data > > > > That will necessarily scan both strings twice, create a new object > > (allocating enough memory to hold them both), then perform a copy operation > > for both. Bottom line: this implementation scales O(n^2) for the size of > > the buffer. > > > > The implementation I gave scales O(n). See: > > > > http://pastie.caboo.se/30456 > > > > Yes, I see the point, you''re using the array to save the multiple > allocations and paying a little extra in the scan. Have you profiled it? > Would it be easy to adapt this for use with protocols like HTTP and SIP, > that alternate newline-delimited lines with data blocks that are delimited > either by a predetermined length or by the end of the stream? More > practically, can you implement EventMachine::Protocols::LineAndTextProtocol > with BufferedTokenizer? > > > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk > >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070116/210d295a/attachment-0001.html
On 1/16/07, Tony Arcieri <tony at clickcaster.com> wrote:> > So, sorry about the delay on getting back to you on this... > > Looking at LineAndTextProtocol, I''m wondering why it takes a buffer size. > Why not just fall back on the typical pattern for reading binary data, i.e. > it''s made available as it''s read. If the desired read buffer is exceeded, > you could have a #fill method to dump data back into the tokenizer''s buffer. > > > Replacing the line parser was fairly simple, but I''m confused as to why > its API is designed in that way. Does the HTTP code request the entire > Content-Length (which is then buffered), or just small chunks at a time?You''re right, I never thought about that. The design implicitly assumes that when you go into "text" (binary) mode, that the size of the binary data is known beforehand, and, crucially, is determined in the course of processing the "line" elements. This is the right way to deal with just about any protocol I can think of that mixes lines with binary data, including HTTP, HTTP chunked-transfers, SIP, etc, etc. (SMTP works exclusively with line-oriented text, and evolved the whole subgenre of content-type encodings in order not to break this rule.) The special-case of HTTP 1.0 is covered by allowing the end of a connection to signal the end of the binary block. More modern protocols like AMQP explicitly require that the size of incoming binary blocks be predetermined, to enable performance optimizations. What you''re contemplating is to allow the end of a binary block to be recognized within the processing of the binary block itself. It ought to be possible to extend the LineAndText handler to do this. I was trying to think of what protocols you might use this for. I thought of protocols that are built up from XML stanzas, like some of the protocols you might use with Jabber and Stomp. Can you think of any others? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070117/9d0cbb7d/attachment.html
My real issue isn''t so much that the API restricts potential protocols than that it''s buffering more than it needs to. At present do you split up binary downloads into fixed sized "chunks" of binary data (e.g. 4096 bytes) at a time and then however much is leftover (e.g. filesize modulo 4096), or does the implementation buffer the entire file before emitting it? I just see the intermediate buffering as being something which can cause potential problems if people don''t think about it. - Tony On 1/17/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> > On 1/16/07, Tony Arcieri <tony at clickcaster.com> wrote: > > > > So, sorry about the delay on getting back to you on this... > > > > Looking at LineAndTextProtocol, I''m wondering why it takes a buffer > > size. Why not just fall back on the typical pattern for reading binary > > data, i.e. it''s made available as it''s read. If the desired read buffer > > is exceeded, you could have a #fill method to dump data back into the > > tokenizer''s buffer. > > > > Replacing the line parser was fairly simple, but I''m confused as to why > > its API is designed in that way. Does the HTTP code request the entire > > Content-Length (which is then buffered), or just small chunks at a time? > > > > > You''re right, I never thought about that. The design implicitly assumes > that when you go into "text" (binary) mode, that the size of the binary data > is known beforehand, and, crucially, is determined in the course of > processing the "line" elements. > > This is the right way to deal with just about any protocol I can think of > that mixes lines with binary data, including HTTP, HTTP chunked-transfers, > SIP, etc, etc. (SMTP works exclusively with line-oriented text, and evolved > the whole subgenre of content-type encodings in order not to break this > rule.) The special-case of HTTP 1.0 is covered by allowing the end of a > connection to signal the end of the binary block. More modern protocols like > AMQP explicitly require that the size of incoming binary blocks be > predetermined, to enable performance optimizations. > > What you''re contemplating is to allow the end of a binary block to be > recognized within the processing of the binary block itself. It ought to be > possible to extend the LineAndText handler to do this. I was trying to think > of what protocols you might use this for. I thought of protocols that are > built up from XML stanzas, like some of the protocols you might use with > Jabber and Stomp. Can you think of any others? > > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk > >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070117/2f8b76e9/attachment.html
On 1/17/07, Tony Arcieri <tony at clickcaster.com> wrote:> > My real issue isn''t so much that the API restricts potential protocols > than that it''s buffering more than it needs to. At present do you split up > binary downloads into fixed sized "chunks" of binary data ( e.g. 4096 > bytes) at a time and then however much is leftover (e.g. filesize modulo > 4096), or does the implementation buffer the entire file before emitting it? > > I just see the intermediate buffering as being something which can cause > potential problems if people don''t think about it.The read buffering is done inside the EM core and is irrespective of how the protocol handlers deal with it. As a general rule EM tries to send as much data to the protocol handler as it can send in one call. This minimizes the transits through the event-dispatch logic, at the expense of the protocol handler having more data to deal with. I hope I understood your comment correctly. Do you think we would benefit from a different strategy? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070117/7ac5a58c/attachment.html
Sorry, I guess I omitted an important detail I was talking specifically about how the HTTP protocol implementation sitting on top of LineAndTextProtocol functions It was one of those "too lazy to look at the code" comments, but that''d probably be the easiest way to answer the question - Tony On 1/17/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> > On 1/17/07, Tony Arcieri <tony at clickcaster.com> wrote: > > > > My real issue isn''t so much that the API restricts potential protocols > > than that it''s buffering more than it needs to. At present do you split up > > binary downloads into fixed sized "chunks" of binary data ( e.g. 4096 > > bytes) at a time and then however much is leftover (e.g. filesize modulo > > 4096), or does the implementation buffer the entire file before emitting it? > > > > I just see the intermediate buffering as being something which can cause > > potential problems if people don''t think about it. > > > > The read buffering is done inside the EM core and is irrespective of how > the protocol handlers deal with it. As a general rule EM tries to send as > much data to the protocol handler as it can send in one call. This minimizes > the transits through the event-dispatch logic, at the expense of the > protocol handler having more data to deal with. I hope I understood your > comment correctly. Do you think we would benefit from a different strategy? > > > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk > >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070117/f3bd676d/attachment.html