I''m working with someone who is having issues with a Transfer- Encoding: Chunked upload. With out a content-length set in the request, everything is treating the content length as 0 and the file is more or less getting sent to the bit bucket. Does anyone have any experience with this kind of issue? We started proxying with nginx(returned a HTTP Error 411), then we tried to directly expose a mongrel and ended up getting strange results, and errors in the mongrel logs. The client is uploading with a mobile phone using J2ME. Zachary Roetemeyer
> With out a content-length set in the > request, everything is treating the content length as 0 and the file > is more or less getting sent to the bit bucket.Is the content length header really not getting set? Mongrel automatically sets this. If you do something like while writing the response code. You might have to overload mongrel''s HttpResponse#send_status method to explicitly not send this header. I think http://merb.devjavu.com/ticket/121 might have some useful hints too. ry
On Feb 15, 2008 9:53 PM, ry dahl <ry at tinyclouds.org> wrote:> > With out a content-length set in the > > request, everything is treating the content length as 0 and the file > > is more or less getting sent to the bit bucket. > > I think http://merb.devjavu.com/ticket/121 might have some useful hints too. >Oh, got lost, there is merb.devjavu.com and also merb.lighthouseapp.com ? :-P -- Luis Lavena Multimedia systems - A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools. Douglas Adams
Looks like mongrel can''t handle chunked posts (see http://mongrel.rubyforge.org/svn/trunk/lib/mongrel/http_request.rb line 29). You''ll need to put another web server (other than nginx) in front of it to handle this, or you could file a bug in the mongrel tracker to get them to add this support (or you could code it up yourself :). On Feb 16, 2008, at 4:35 AM, Zachary Roetemeyer wrote:> I''m working with someone who is having issues with a Transfer- > Encoding: Chunked upload. With out a content-length set in the > request, everything is treating the content length as 0 and the file > is more or less getting sent to the bit bucket. Does anyone have any > experience with this kind of issue? We started proxying with > nginx(returned a HTTP Error 411), then we tried to directly expose a > mongrel and ended up getting strange results, and errors in the > mongrel logs. The client is uploading with a mobile phone using J2ME. > > Zachary Roetemeyer > > > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
On Fri, 15 Feb 2008 14:35:04 -0600 Zachary Roetemeyer <zprime at gmail.com> wrote:> I''m working with someone who is having issues with a Transfer- > Encoding: Chunked upload. With out a content-length set in the > request, everything is treating the content length as 0 and the file > is more or less getting sent to the bit bucket. Does anyone have any > experience with this kind of issue? We started proxying with > nginx(returned a HTTP Error 411), then we tried to directly expose a > mongrel and ended up getting strange results, and errors in the > mongrel logs. The client is uploading with a mobile phone using J2ME.Uh, you aren''t allowed to set a Content-Length AND use CE in the same request. That would let the two contradict so it''s an RFC violation. Actually, WebLogic makes this mistake and we had to school them about it until they told us to turn it off. (WebLogic is such a piece of shit). My RFC knowledge on CE is rusty since I banished that shitty part of HTTP from my mind when I implemented it in RFuzz, but my thoughts are: 1) CE wasn''t designed for clients to upload, since no browser does it. 2) It''s originally for a server to send periodic parts of a request without having to know it''s total final size. 3) Anyone doing a CE as a client is seriously screwing with the RFC and really exercising the corners of it for no apparent reason since CE isn''t needed by a client (the client knows the size, or better). 4) This puts a burden on the server to process the CE as it comes over, which has the same problems as mime boundaries but with a more horribly designed part of the RFC. 5) My recommendation is that you just don''t do this. The value add for a client (or even modern servers really) is quite minimal. Now, how you go about implementing it: A) Read up on how to write a Mongrel handler. B) Register it at the front of your web app or create a separate server that runs just mongrel (probably your best if you want any kind of performance). C) Your handler would detect the CE via the normal headers and such, and then process the final request body as it comes in for each chunk. D) This might involve subclassing the Http processing parts that handle the body and doing a bit of other hacking, and it''s seriously fucking dumb because after that you''ve gotta process mime boundaries and other stuff, which will punish your server heavily. Enjoy! -- Zed A. Shaw - Hate: http://savingtheinternetwithhate.com/ - Good: http://www.zedshaw.com/ - Evil: http://yearofevil.com/
The upload is coming from a mobile device with J2ME, and it does this by default with anything over 2K, and apparently there''s no way to turn it off. On Feb 17, 2008, at 8:18 PM, Zed A. Shaw wrote:> On Fri, 15 Feb 2008 14:35:04 -0600 > Zachary Roetemeyer <zprime at gmail.com> wrote: > >> I''m working with someone who is having issues with a Transfer- >> Encoding: Chunked upload. With out a content-length set in the >> request, everything is treating the content length as 0 and the file >> is more or less getting sent to the bit bucket. Does anyone have any >> experience with this kind of issue? We started proxying with >> nginx(returned a HTTP Error 411), then we tried to directly expose a >> mongrel and ended up getting strange results, and errors in the >> mongrel logs. The client is uploading with a mobile phone using >> J2ME. > > Uh, you aren''t allowed to set a Content-Length AND use CE in the same > request. That would let the two contradict so it''s an RFC violation. > Actually, WebLogic makes this mistake and we had to school them about > it until they told us to turn it off. (WebLogic is such a piece of > shit). > > My RFC knowledge on CE is rusty since I banished that shitty part of > HTTP from my mind when I implemented it in RFuzz, but my thoughts are: > > 1) CE wasn''t designed for clients to upload, since no browser does it. > 2) It''s originally for a server to send periodic parts of a request > without having to know it''s total final size. > 3) Anyone doing a CE as a client is seriously screwing with the RFC > and > really exercising the corners of it for no apparent reason since CE > isn''t needed by a client (the client knows the size, or better). > 4) This puts a burden on the server to process the CE as it comes > over, which has the same problems as mime boundaries but with a more > horribly designed part of the RFC. > 5) My recommendation is that you just don''t do this. The value add > for > a client (or even modern servers really) is quite minimal. > > Now, how you go about implementing it: > > A) Read up on how to write a Mongrel handler. > B) Register it at the front of your web app or create a separate > server > that runs just mongrel (probably your best if you want any kind of > performance). > C) Your handler would detect the CE via the normal headers and such, > and then process the final request body as it comes in for each chunk. > D) This might involve subclassing the Http processing parts that > handle > the body and doing a bit of other hacking, and it''s seriously fucking > dumb because after that you''ve gotta process mime boundaries and other > stuff, which will punish your server heavily. > > Enjoy! > > -- > Zed A. Shaw > - Hate: http://savingtheinternetwithhate.com/ > - Good: http://www.zedshaw.com/ > - Evil: http://yearofevil.com/ > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-usersZachary Roetemeyer zprime at gmail.com
I implemented a chunked transfer encoding to handle ESI in pages... Each fragment is sent as a chunk... Since, I can''t determine ahead of time what the content-length would be as the individual ESI fragments are of unknown length until the page is assembled... I can see where a mobile device might want to do this because of memory constraints, however.... if the device has what''s it''s sending stored somewhere one would hope it can measure that something''s length, ahead of time - writing the Content-length first... Then connect the write end of the socket to the read end of the file... and away it goes never storing more then the OS decides to keep in memory... This page might be helpful... http://www.jmarshall.com/easy/http/#http1.1c2 On Feb 18, 2008 7:39 AM, Zachary Roetemeyer <zprime at gmail.com> wrote:> The upload is coming from a mobile device with J2ME, and it does this > by default with anything over 2K, and apparently there''s no way to > turn it off. > > On Feb 17, 2008, at 8:18 PM, Zed A. Shaw wrote: > > > On Fri, 15 Feb 2008 14:35:04 -0600 > > Zachary Roetemeyer <zprime at gmail.com> wrote: > > > >> I''m working with someone who is having issues with a Transfer- > >> Encoding: Chunked upload. With out a content-length set in the > >> request, everything is treating the content length as 0 and the file > >> is more or less getting sent to the bit bucket. Does anyone have any > >> experience with this kind of issue? We started proxying with > >> nginx(returned a HTTP Error 411), then we tried to directly expose a > >> mongrel and ended up getting strange results, and errors in the > >> mongrel logs. The client is uploading with a mobile phone using > >> J2ME. > > > > Uh, you aren''t allowed to set a Content-Length AND use CE in the same > > request. That would let the two contradict so it''s an RFC violation. > > Actually, WebLogic makes this mistake and we had to school them about > > it until they told us to turn it off. (WebLogic is such a piece of > > shit). > > > > My RFC knowledge on CE is rusty since I banished that shitty part of > > HTTP from my mind when I implemented it in RFuzz, but my thoughts are: > > > > 1) CE wasn''t designed for clients to upload, since no browser does it. > > 2) It''s originally for a server to send periodic parts of a request > > without having to know it''s total final size. > > 3) Anyone doing a CE as a client is seriously screwing with the RFC > > and > > really exercising the corners of it for no apparent reason since CE > > isn''t needed by a client (the client knows the size, or better). > > 4) This puts a burden on the server to process the CE as it comes > > over, which has the same problems as mime boundaries but with a more > > horribly designed part of the RFC. > > 5) My recommendation is that you just don''t do this. The value add > > for > > a client (or even modern servers really) is quite minimal. > > > > Now, how you go about implementing it: > > > > A) Read up on how to write a Mongrel handler. > > B) Register it at the front of your web app or create a separate > > server > > that runs just mongrel (probably your best if you want any kind of > > performance). > > C) Your handler would detect the CE via the normal headers and such, > > and then process the final request body as it comes in for each chunk. > > D) This might involve subclassing the Http processing parts that > > handle > > the body and doing a bit of other hacking, and it''s seriously fucking > > dumb because after that you''ve gotta process mime boundaries and other > > stuff, which will punish your server heavily. > > > > Enjoy! > > > > -- > > Zed A. Shaw > > - Hate: http://savingtheinternetwithhate.com/ > > - Good: http://www.zedshaw.com/ > > - Evil: http://yearofevil.com/ > > _______________________________________________ > > Mongrel-users mailing list > > Mongrel-users at rubyforge.org > > http://rubyforge.org/mailman/listinfo/mongrel-users > > Zachary Roetemeyer > zprime at gmail.com > > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://rubyforge.org/pipermail/mongrel-users/attachments/20080218/5d8790df/attachment.html
Mind posting a patch? As an aside, the j2me people seem to think CE requests are part the http 1.1 spec (rfc2616): http://developers.sun.com/mobility/midp/questions/chunking/ and the minimal justification they give is that "interactive applications may not know ahead of time how much data they''re going to send" (paraphrase). On Feb 19, 2008, at 4:44 AM, Todd Fisher wrote:> I implemented a chunked transfer encoding to handle ESI in pages... > Each fragment is sent as a chunk... Since, I can''t determine ahead > of time what the content-length would be as the individual ESI > fragments are of unknown length until the page is assembled... > > I can see where a mobile device might want to do this because of > memory constraints, however.... if the device has what''s it''s > sending stored somewhere one would hope it can measure that > something''s length, ahead of time - writing the Content-length > first... Then connect the write end of the socket to the read end of > the file... and away it goes never storing more then the OS > decides to keep in memory... > > This page might be helpful... http://www.jmarshall.com/easy/http/#http1.1c2 > > > > On Feb 18, 2008 7:39 AM, Zachary Roetemeyer <zprime at gmail.com> wrote: > The upload is coming from a mobile device with J2ME, and it does this > by default with anything over 2K, and apparently there''s no way to > turn it off. > > On Feb 17, 2008, at 8:18 PM, Zed A. Shaw wrote: > > > On Fri, 15 Feb 2008 14:35:04 -0600 > > Zachary Roetemeyer <zprime at gmail.com> wrote: > > > >> I''m working with someone who is having issues with a Transfer- > >> Encoding: Chunked upload. With out a content-length set in the > >> request, everything is treating the content length as 0 and the > file > >> is more or less getting sent to the bit bucket. Does anyone have > any > >> experience with this kind of issue? We started proxying with > >> nginx(returned a HTTP Error 411), then we tried to directly > expose a > >> mongrel and ended up getting strange results, and errors in the > >> mongrel logs. The client is uploading with a mobile phone using > >> J2ME. > > > > Uh, you aren''t allowed to set a Content-Length AND use CE in the > same > > request. That would let the two contradict so it''s an RFC > violation. > > Actually, WebLogic makes this mistake and we had to school them > about > > it until they told us to turn it off. (WebLogic is such a piece of > > shit). > > > > My RFC knowledge on CE is rusty since I banished that shitty part of > > HTTP from my mind when I implemented it in RFuzz, but my thoughts > are: > > > > 1) CE wasn''t designed for clients to upload, since no browser does > it. > > 2) It''s originally for a server to send periodic parts of a request > > without having to know it''s total final size. > > 3) Anyone doing a CE as a client is seriously screwing with the RFC > > and > > really exercising the corners of it for no apparent reason since CE > > isn''t needed by a client (the client knows the size, or better). > > 4) This puts a burden on the server to process the CE as it comes > > over, which has the same problems as mime boundaries but with a more > > horribly designed part of the RFC. > > 5) My recommendation is that you just don''t do this. The value add > > for > > a client (or even modern servers really) is quite minimal. > > > > Now, how you go about implementing it: > > > > A) Read up on how to write a Mongrel handler. > > B) Register it at the front of your web app or create a separate > > server > > that runs just mongrel (probably your best if you want any kind of > > performance). > > C) Your handler would detect the CE via the normal headers and such, > > and then process the final request body as it comes in for each > chunk. > > D) This might involve subclassing the Http processing parts that > > handle > > the body and doing a bit of other hacking, and it''s seriously > fucking > > dumb because after that you''ve gotta process mime boundaries and > other > > stuff, which will punish your server heavily. > > > > Enjoy! > > > > -- > > Zed A. Shaw > > - Hate: http://savingtheinternetwithhate.com/ > > - Good: http://www.zedshaw.com/ > > - Evil: http://yearofevil.com/ > > _______________________________________________ > > Mongrel-users mailing list > > Mongrel-users at rubyforge.org > > http://rubyforge.org/mailman/listinfo/mongrel-users > > Zachary Roetemeyer > zprime at gmail.com > > > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users > > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
Zachary, mind posting some of the code you''re using in J2ME? I''ve got no dog in this fight. On Feb 20, 2008, at 12:55 PM, Zed A. Shaw wrote:> On Tue, 19 Feb 2008 08:36:43 +0800 > Eden Li <eden at mojiti.com> wrote: > >> Mind posting a patch? >> >> As an aside, the j2me people seem to think CE requests are part the >> http 1.1 spec (rfc2616): >> >> http://developers.sun.com/mobility/midp/questions/chunking/ >> >> and the minimal justification they give is that "interactive >> applications may not know ahead of time how much data they''re going >> to >> send" (paraphrase). > > Alright, let''s see the code being used. If this is truly the only way > to submit a request from a memory constrained device, and I can''t find > an alternative that''s simpler then I''ll add CE to the Mongrel handler. > CE is already in the RFuzz handler so it''d be nothing more than > putting > the same code in the Mongrel parser. Of course this will just be an > exercise and I''ll let the mongrel crew decide if they want it in or > not. > > And when I say "see the code" I don''t mean a toy version, I mean the > actual code being used to do the upload that I can work with and test. > Doesn''t have to be the full app, but enough for me to run in a MIDP > emulator and try out. > > -- > Zed A. Shaw > - Hate: http://savingtheinternetwithhate.com/ > - Good: http://www.zedshaw.com/ > - Evil: http://yearofevil.com/ > _______________________________________________ > Mongrel-users mailing list > Mongrel-users at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-users
On Tue, 19 Feb 2008 08:36:43 +0800 Eden Li <eden at mojiti.com> wrote:> Mind posting a patch? > > As an aside, the j2me people seem to think CE requests are part the > http 1.1 spec (rfc2616): > > http://developers.sun.com/mobility/midp/questions/chunking/ > > and the minimal justification they give is that "interactive > applications may not know ahead of time how much data they''re going to > send" (paraphrase).Alright, let''s see the code being used. If this is truly the only way to submit a request from a memory constrained device, and I can''t find an alternative that''s simpler then I''ll add CE to the Mongrel handler. CE is already in the RFuzz handler so it''d be nothing more than putting the same code in the Mongrel parser. Of course this will just be an exercise and I''ll let the mongrel crew decide if they want it in or not. And when I say "see the code" I don''t mean a toy version, I mean the actual code being used to do the upload that I can work with and test. Doesn''t have to be the full app, but enough for me to run in a MIDP emulator and try out. -- Zed A. Shaw - Hate: http://savingtheinternetwithhate.com/ - Good: http://www.zedshaw.com/ - Evil: http://yearofevil.com/
I''ll have to see what I can do since the code isn''t mine. On Feb 19, 2008, at 9:03 PM, Eden Li wrote:> Zachary, mind posting some of the code you''re using in J2ME? I''ve > got no dog in this fight. > > On Feb 20, 2008, at 12:55 PM, Zed A. Shaw wrote: > >> On Tue, 19 Feb 2008 08:36:43 +0800 >> Eden Li <eden at mojiti.com> wrote: >> >>> Mind posting a patch? >>> >>> As an aside, the j2me people seem to think CE requests are part the >>> http 1.1 spec (rfc2616): >>> >>> http://developers.sun.com/mobility/midp/questions/chunking/ >>> >>> and the minimal justification they give is that "interactive >>> applications may not know ahead of time how much data they''re >>> going to >>> send" (paraphrase). >> >> Alright, let''s see the code being used. If this is truly the only >> way >> to submit a request from a memory constrained device, and I can''t >> find >> an alternative that''s simpler then I''ll add CE to the Mongrel >> handler. >> CE is already in the RFuzz handler so it''d be nothing more than >> putting >> the same code in the Mongrel parser. Of course this will just be an >> exercise and I''ll let the mongrel crew decide if they want it in or >> not. >> >> And when I say "see the code" I don''t mean a toy version, I mean the >> actual code being used to do the upload that I can work with and >> test. >> Doesn''t have to be the full app, but enough for me to run in a MIDP >> emulator and try out. >> >> -- >> Zed A. Shaw >> - Hate: http://savingtheinternetwithhate.com/ >> - Good: http://www.zedshaw.com/ >> - Evil: http://yearofevil.com/ >> _______________________________________________ >> Mongrel-users mailing list >> Mongrel-users at rubyforge.org >> http://rubyforge.org/mailman/listinfo/mongrel-users >Zachary Roetemeyer zprime at gmail.com