I noticed EventMachine::Protocols::HttpClient sends an HTTP/1.1 request but doesn''t send "Connection: close" in the request. Per the RFC: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html> HTTP/1.1 applications that do not support persistent connections MUSTinclude the "close" connection option in every message. The problem occurs specifically when the server does not send a Content-Length in the response. EventMachine''s behavior in this case is to wait until the remote server closes the connection before it dispatches the response. However, since "Connection: close" isn''t getting sent in the initial request, the default HTTP/1.1 behavior is to keep the connection alive, so EventMachine just sits there waiting until eventually the server''s keep-alive timeout is reached. So a request that has completed waits the full duration of the remote server''s keep-alive timeout before dispatching a response. Also, this is a bit pedantic, but any reason you aren''t capitalizing the request headers the same way the RFC does (e.g. Like-This) -- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070227/780954c9/attachment.html
Also note: I think the best solution to this problem is to have HttpClient send an HTTP/1.0 request until proper HTTP/1.1 support is implemented. Right now it''s behaving like an HTTP/1.0 client. - Tony On 2/27/07, Tony Arcieri <tony at clickcaster.com> wrote:> > I noticed EventMachine::Protocols::HttpClient sends an HTTP/1.1 request > but doesn''t send "Connection: close" in the request. Per the RFC: > > http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html > > > HTTP/1.1 applications that do not support persistent connections MUST > include the "close" connection option in every message. > > The problem occurs specifically when the server does not send a > Content-Length in the response. EventMachine''s behavior in this case is to > wait until the remote server closes the connection before it dispatches the > response. However, since "Connection: close" isn''t getting sent in the > initial request, the default HTTP/1.1 behavior is to keep the connection > alive, so EventMachine just sits there waiting until eventually the server''s > keep-alive timeout is reached. So a request that has completed waits the > full duration of the remote server''s keep-alive timeout before dispatching a > response. > > Also, this is a bit pedantic, but any reason you aren''t capitalizing the > request headers the same way the RFC does (e.g. Like-This) > > -- > Tony Arcieri > ClickCaster, Inc. > tony at clickcaster.com > (970) 232-4208-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070227/c4ecbaed/attachment-0001.html
And any chance of getting support for the Range header? I mean, all these changes are trivial, I could make them myself and check it in. - Tony On 2/27/07, Tony Arcieri <tony at clickcaster.com> wrote:> > Also note: I think the best solution to this problem is to have HttpClient > send an HTTP/1.0 request until proper HTTP/1.1 support is implemented. > Right now it''s behaving like an HTTP/1.0 client. > > - Tony > > On 2/27/07, Tony Arcieri <tony at clickcaster.com> wrote: > > > > I noticed EventMachine::Protocols::HttpClient sends an HTTP/1.1 request > > but doesn''t send "Connection: close" in the request. Per the RFC: > > > > http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html > > > > > HTTP/1.1 applications that do not support persistent connections MUST > > include the "close" connection option in every message. > > > > The problem occurs specifically when the server does not send a > > Content-Length in the response. EventMachine''s behavior in this case is to > > wait until the remote server closes the connection before it dispatches the > > response. However, since "Connection: close" isn''t getting sent in the > > initial request, the default HTTP/1.1 behavior is to keep the connection > > alive, so EventMachine just sits there waiting until eventually the server''s > > keep-alive timeout is reached. So a request that has completed waits the > > full duration of the remote server''s keep-alive timeout before dispatching a > > response. > > > > Also, this is a bit pedantic, but any reason you aren''t capitalizing the > > request headers the same way the RFC does (e.g. Like-This) > > > > -- > > Tony Arcieri > > ClickCaster, Inc. > > tony at clickcaster.com > > (970) 232-4208 > > > > > -- > Tony Arcieri > ClickCaster, Inc. > tony at clickcaster.com > (970) 232-4208 >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070227/fa1f2d94/attachment.html
I went ahead and flipped HTTP/1.1 to HTTP/1.0 in the request httpclient.rbsends and checked that in. Let me know if that''s a problem. - Tony On 2/27/07, Tony Arcieri <tony at clickcaster.com> wrote:> > And any chance of getting support for the Range header? > > I mean, all these changes are trivial, I could make them myself and check > it in. > > - Tony > > On 2/27/07, Tony Arcieri <tony at clickcaster.com> wrote: > > > > Also note: I think the best solution to this problem is to have > > HttpClient send an HTTP/1.0 request until proper HTTP/1.1 support is > > implemented. Right now it''s behaving like an HTTP/1.0 client. > > > > - Tony > > > > On 2/27/07, Tony Arcieri <tony at clickcaster.com > wrote: > > > > > > I noticed EventMachine::Protocols::HttpClient sends an HTTP/1.1 > > > request but doesn''t send "Connection: close" in the request. Per the RFC: > > > > > > http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html > > > > > > > HTTP/1.1 applications that do not support persistent connections > > > MUST include the "close" connection option in every message. > > > > > > The problem occurs specifically when the server does not send a > > > Content-Length in the response. EventMachine''s behavior in this case is to > > > wait until the remote server closes the connection before it dispatches the > > > response. However, since "Connection: close" isn''t getting sent in the > > > initial request, the default HTTP/1.1 behavior is to keep the connection > > > alive, so EventMachine just sits there waiting until eventually the server''s > > > keep-alive timeout is reached. So a request that has completed waits the > > > full duration of the remote server''s keep-alive timeout before dispatching a > > > response. > > > > > > Also, this is a bit pedantic, but any reason you aren''t capitalizing > > > the request headers the same way the RFC does (e.g. Like-This) > > > > > > -- > > > Tony Arcieri > > > ClickCaster, Inc. > > > tony at clickcaster.com > > > (970) 232-4208 > > > > > > > > > > -- > > Tony Arcieri > > ClickCaster, Inc. > > tony at clickcaster.com > > (970) 232-4208 > > > > > > -- > Tony Arcieri > ClickCaster, Inc. > tony at clickcaster.com > (970) 232-4208 >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070228/ae1a2bf5/attachment.html
On 2/28/07, Tony Arcieri <tony at clickcaster.com> wrote:> > I went ahead and flipped HTTP/1.1 to HTTP/1.0 in the request httpclient.rbsends and checked that in. Let me know if that''s a problem.Here''s the rationale for the lower-case header names, such as it is: The RFC (2046, actually) requires that header-names be treated case-insensitively, but it''s always convenient to put them into Ruby hashes keyed by the header name. So I''ve gotten into the habit of downcasing them before storing them in a hash. I''m not seeing the necessity of sending an HTTP/1.0 request. If a server (even an HTTP/1.1 server) chooses not to send a Content-length header, then the RFC requires it to close the connection when all the data has been sent. (And HttpClient will pick that up by handling the EM unbound event.) Exceptions would be if the server sends a chunked response or a multipart/mixed response where the client can see the end of the stream without reference to Content-length. HttpClient doesn''t currently handle either of those cases at this point. Is that perhaps what happened in your case? Also, there''s no reason why we can''t implement persistent connections in HttpClient. Maybe have an additional request flag that would signal not to close the connection unless the server does. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070228/859c1b89/attachment-0001.html
You''re mistaken in regard to HTTP/1.1 requiring the connection be closed unless a Content-Length is set: http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4 While closing the connection is one of the ways that a server can signal the end of content, chunked transfer encoding can be used, in which case the connection will be left open. I ran into this case specifically on FeedBurner. Try to grab this with HttpClient: http://feeds.feedburner.com/theshowwithzefrank However, the RFC states explicitly that unless persistent connections are supported, the client must send Connection: close. Even so, the HttpClient implementation should also support chunked transfer encoding before it sends an HTTP/1.1 request. I''d say the various options are: 1) Send an HTTP/1.0 request 2) Implement support for chunked transfer encoding and send Connection: close with an HTTP/1.1 request 3) Implement support for both chunked transfer encoding and persistent connections and send an HTTP/1.1 request I opted for #1, the simplest solution. -- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 On 2/28/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> > On 2/28/07, Tony Arcieri <tony at clickcaster.com> wrote: > > > > I went ahead and flipped HTTP/1.1 to HTTP/1.0 in the request > > httpclient.rb sends and checked that in. Let me know if that''s a > > problem. > > > > Here''s the rationale for the lower-case header names, such as it is: The > RFC (2046, actually) requires that header-names be treated > case-insensitively, but it''s always convenient to put them into Ruby hashes > keyed by the header name. So I''ve gotten into the habit of downcasing them > before storing them in a hash. > > I''m not seeing the necessity of sending an HTTP/1.0 request. If a server > (even an HTTP/1.1 server) chooses not to send a Content-length header, then > the RFC requires it to close the connection when all the data has been sent. > (And HttpClient will pick that up by handling the EM unbound event.) > Exceptions would be if the server sends a chunked response or a > multipart/mixed response where the client can see the end of the stream > without reference to Content-length. HttpClient doesn''t currently handle > either of those cases at this point. Is that perhaps what happened in your > case? > > Also, there''s no reason why we can''t implement persistent connections in > HttpClient. Maybe have an additional request flag that would signal not to > close the connection unless the server does. > > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070228/f6cd180f/attachment.html
On 2/28/07, Tony Arcieri <tony at clickcaster.com> wrote:> > You''re mistaken in regard to HTTP/1.1 requiring the connection be closed > unless a Content-Length is set: > > http://www.w3.org/Protocols/rfc2616/rfc2616-sgeec4.html#sec4.4 > <http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4> > > While closing the connection is one of the ways that a server can signal > the end of content, chunked transfer encoding can be used, in which case the > connection will be left open. I ran into this case specifically on > FeedBurner. Try to grab this with HttpClient: > > http://feeds.feedburner.com/theshowwithzefrank > > However, the RFC states explicitly that unless persistent connections are > supported, the client must send Connection: close. Even so, the HttpClient > implementation should also support chunked transfer encoding before it sends > an HTTP/1.1 request. > > I''d say the various options are: > > 1) Send an HTTP/1.0 request > 2) Implement support for chunked transfer encoding and send Connection: > close with an HTTP/1.1 request > 3) Implement support for both chunked transfer encoding and persistent > connections and send an HTTP/1.1 request > > I opted for #1, the simplest solution. > > -- > Tony Arcieri > ClickCaster, Inc. > tony at clickcaster.com > (970) 232-4208 > > On 2/28/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote: > > > On 2/28/07, Tony Arcieri <tony at clickcaster.com> wrote: > > > > > > I went ahead and flipped HTTP/1.1 to HTTP/1.0 in the request > > > httpclient.rb sends and checked that in. Let me know if that''s a > > > problem. > > > > > > > > Here''s the rationale for the lower-case header names, such as it is: The > > RFC (2046, actually) requires that header-names be treated > > case-insensitively, but it''s always convenient to put them into Ruby hashes > > keyed by the header name. So I''ve gotten into the habit of downcasing them > > before storing them in a hash. > > > > I''m not seeing the necessity of sending an HTTP/1.0 request. If a server > > (even an HTTP/1.1 server) chooses not to send a Content-length header, then > > the RFC requires it to close the connection when all the data has been sent. > > (And HttpClient will pick that up by handling the EM unbound event.) > > Exceptions would be if the server sends a chunked response or a > > multipart/mixed response where the client can see the end of the stream > > without reference to Content-length. HttpClient doesn''t currently handle > > either of those cases at this point. Is that perhaps what happened in your > > case? > > > > Also, there''s no reason why we can''t implement persistent connections in > > HttpClient. Maybe have an additional request flag that would signal not to > > close the connection unless the server does. > > > > _______________________________________________ > > Eventmachine-talk mailing list > > Eventmachine-talk at rubyforge.org > > http://rubyforge.org/mailman/listinfo/eventmachine-talk > > > > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070228/7a3e11e0/attachment-0001.html
On 2/28/07, Tony Arcieri <tony at clickcaster.com> wrote:> > You''re mistaken in regard to HTTP/1.1 requiring the connection be closed > unless a Content-Length is set: > > http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4 > > While closing the connection is one of the ways that a server can signal > the end of content, chunked transfer encoding can be used, in which case the > connection will be left open. I ran into this case specifically on > FeedBurner. Try to grab this with HttpClient: > > http://feeds.feedburner.com/theshowwithzefrank > > However, the RFC states explicitly that unless persistent connections are > supported, the client must send Connection: close. Even so, the HttpClient > implementation should also support chunked transfer encoding before it sends > an HTTP/1.1 request. > > I''d say the various options are: > > 1) Send an HTTP/1.0 request > 2) Implement support for chunked transfer encoding and send Connection: > close with an HTTP/1.1 request > 3) Implement support for both chunked transfer encoding and persistent > connections and send an HTTP/1.1 request > > I opted for #1, the simplest solution.Ok, I found the language in the RFC 2616 requiring an "application" to send Connection: close if it doesn''t support persistent connections. It''s in graff 14.10, not 4.4. As I said earlier, I suspected your problem had to do with a chunked transfer from the server. It''s ok for to step this code back to 1.0, but it would be just as easy to add Connection: close. I think this would be better because in this case we''re signaling to the server that the *client* will close the connection, not the other way around, which is far less scalable. Some really stupid web servers, like IIS, might also do undesirable things with HTTP/1.0 and Connection: close. We MUST support chunked transfers, since they''re the general case these days, with documents above a certain size. And I''d like to support persistence as well. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070228/881f702b/attachment.html
Chunked transfers definitely should be supported, and are the next thing that should be implemented in the HttpClient library. My needs are satisfied by just sending HTTP/1.0 however. I''m getting great speed (in this case pulling in offsite RSS feeds), getting about 11 feeds a second on average with 16 concurrent connections. - Tony On 2/28/07, Francis Cianfrocca <garbagecat10 at gmail.com> wrote:> > On 2/28/07, Tony Arcieri <tony at clickcaster.com> wrote: > > > > You''re mistaken in regard to HTTP/1.1 requiring the connection be closed > > unless a Content-Length is set: > > > > http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4 > > > > While closing the connection is one of the ways that a server can signal > > the end of content, chunked transfer encoding can be used, in which case the > > connection will be left open. I ran into this case specifically on > > FeedBurner. Try to grab this with HttpClient: > > > > http://feeds.feedburner.com/theshowwithzefrank > > > > However, the RFC states explicitly that unless persistent connections > > are supported, the client must send Connection: close. Even so, the > > HttpClient implementation should also support chunked transfer encoding > > before it sends an HTTP/1.1 request. > > > > I''d say the various options are: > > > > 1) Send an HTTP/1.0 request > > 2) Implement support for chunked transfer encoding and send Connection: > > close with an HTTP/1.1 request > > 3) Implement support for both chunked transfer encoding and persistent > > connections and send an HTTP/1.1 request > > > > I opted for #1, the simplest solution. > > > > Ok, I found the language in the RFC 2616 requiring an "application" to > send Connection: close if it doesn''t support persistent connections. It''s in > graff 14.10, not 4.4. As I said earlier, I suspected your problem had to > do with a chunked transfer from the server. > > It''s ok for to step this code back to 1.0, but it would be just as easy to > add Connection: close. I think this would be better because in this case > we''re signaling to the server that the *client* will close the connection, > not the other way around, which is far less scalable. Some really stupid web > servers, like IIS, might also do undesirable things with HTTP/1.0 and > Connection: close. > > We MUST support chunked transfers, since they''re the general case these > days, with documents above a certain size. And I''d like to support > persistence as well. > > > _______________________________________________ > Eventmachine-talk mailing list > Eventmachine-talk at rubyforge.org > http://rubyforge.org/mailman/listinfo/eventmachine-talk >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070228/536e8e2a/attachment-0001.html
For DistribuStream, I will be needing HTTP/1.1 features in the client library, specifically Range support. In terms of the HTTP client, I''m thinking: Add args[:range] to the request, which can be one of the following: A Range object, which would map n..m to "n-m" A Fixnum, which would translate n to "n-" if n >= 0 or "-n" if n < 0 (-n retrieves the last n bytes of the given resource) An Array containing either of the above. Each would be rendered using the above rules and then joined with "," This would give you the full flexibility of Range requests. Once you''ve constructed the ranges using the above rules, you just add a Range: bytes=<range> to the request header. The server should then send back 206 Partial Content. One of the prerequisites of doing all of this is getting HttpClient to support chunked encoding so (provided Connection: close or persistent connections are implemented) it can function as a proper HTTP/1.1 client. - Tony On 2/28/07, Tony Arcieri <tony at clickcaster.com> wrote:> > Chunked transfers definitely should be supported, and are the next thing > that should be implemented in the HttpClient library. > > My needs are satisfied by just sending HTTP/1.0 however. I''m getting > great speed (in this case pulling in offsite RSS feeds), getting about 11 > feeds a second on average with 16 concurrent connections. > > - Tony > > On 2/28/07, Francis Cianfrocca < garbagecat10 at gmail.com> wrote: > > > > On 2/28/07, Tony Arcieri <tony at clickcaster.com> wrote: > > > > > > You''re mistaken in regard to HTTP/1.1 requiring the connection be > > > closed unless a Content-Length is set: > > > > > > http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4 > > > > > > While closing the connection is one of the ways that a server can > > > signal the end of content, chunked transfer encoding can be used, in which > > > case the connection will be left open. I ran into this case specifically on > > > FeedBurner. Try to grab this with HttpClient: > > > > > > http://feeds.feedburner.com/theshowwithzefrank > > > > > > However, the RFC states explicitly that unless persistent connections > > > are supported, the client must send Connection: close. Even so, the > > > HttpClient implementation should also support chunked transfer encoding > > > before it sends an HTTP/1.1 request. > > > > > > I''d say the various options are: > > > > > > 1) Send an HTTP/1.0 request > > > 2) Implement support for chunked transfer encoding and send > > > Connection: close with an HTTP/1.1 request > > > 3) Implement support for both chunked transfer encoding and persistent > > > connections and send an HTTP/1.1 request > > > > > > I opted for #1, the simplest solution. > > > > > > > > Ok, I found the language in the RFC 2616 requiring an "application" to > > send Connection: close if it doesn''t support persistent connections. It''s in > > graff 14.10, not 4.4. As I said earlier, I suspected your problem had to > > do with a chunked transfer from the server. > > > > It''s ok for to step this code back to 1.0, but it would be just as easy > > to add Connection: close. I think this would be better because in this case > > we''re signaling to the server that the *client* will close the connection, > > not the other way around, which is far less scalable. Some really stupid web > > servers, like IIS, might also do undesirable things with HTTP/1.0 and > > Connection: close. > > > > We MUST support chunked transfers, since they''re the general case these > > days, with documents above a certain size. And I''d like to support > > persistence as well. > > > > > > _______________________________________________ > > Eventmachine-talk mailing list > > Eventmachine-talk at rubyforge.org > > http://rubyforge.org/mailman/listinfo/eventmachine-talk > > > > > > -- > Tony Arcieri > ClickCaster, Inc. > tony at clickcaster.com > (970) 232-4208 >-- Tony Arcieri ClickCaster, Inc. tony at clickcaster.com (970) 232-4208 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/eventmachine-talk/attachments/20070302/3110b0b4/attachment-0001.html