I''ve been trying to understand what happens in Unicorn when a client terminates a connection, and nginx logs a 499 response code. In my debugging this can happen if the client is on a flaky connection, or if they double-click a form submit button, the first request is terminated and nginx logs a 499 response code. It seems that in this case the Rails app actually aborts the request, wherever it is in the course of it. The issue I ran into is that my app made a destructive request to an external service in the context of a request, but the client disconnected before the app was able to respond. So the external service returned its response but the request was aborted before the app was able to commit its transaction to the database, confusion ensued. Can you confirm that this is actually what happens in Unicorn when the client disconnects? I''m not seeing anything in the logs to indicate the actual behaviour. In dealing with this I''m thinking about turning on proxy_ignore_client_abort (http://wiki.nginx.org/HttpProxyModule#proxy_ignore_client_abort) so that requests that make it to the Rails app aren''t aborted. Does anyone have experience with this? I can see it causing its own sorts of confusion. Jesse
Jesse Storimer <jstorimer at gmail.com> wrote:> I''ve been trying to understand what happens in Unicorn when a client > terminates a connection, and nginx logs a 499 response code.We''d have to read the nginx sources to answer what nginx does, but of course I can answer what Unicorn does off the top of my head.> In my debugging this can happen if the client is on a flaky connection, > or if they double-click a form submit button, the first request is > terminated and nginx logs a 499 response code.<snip> (the snipped paragraph deserves independent observation/attention)> Can you confirm that this is actually what happens in Unicorn when the > client disconnects? I''m not seeing anything in the logs to indicate the > actual behaviour.It depends on when exactly the client (nginx) disconnect is detected. Unicorn has 4 distinct states : 1) reading headers, if a client disconnects before it has written _all_ of its request headers, the Rack app will never be called. Since no applicaton logic fired at this point. 2) inside Rack dispatch (rack.input reading) This will abort the Rack application dispatch if your client disconnects before _all_ of the request body is sent. Unlike most servers, Unicorn lazily reads any request bodies. You can catch exceptions from env["rack.input"].{read,gets,each} to detect this. The Unicorn::PrereadInput middleware can minimize the time window for this state by reading the request body ASAP. You can also ignore this if your app isn''t handling requests with bodies (POST/PUT), but since you mentioned form input... 3) inside Rack dispatch (after rack.input reading) Your app has no way of knowing your client disconnected at this stage. You can hack Unicorn to IO.select in a separate thread, but there''ll always be exposed windows leading up to 4) so it''s not worth it... 4) writing the response: Unicorn will abort whenever a socket error is detected. Keep in mind that every single part of the Rack response array can be dynamically generated by the app. Your application can still be "running" even though the Rack app has returned its response for Unicorn to start writing. Clients/Rack middleware can be written to detect this in the response body "close" method by checking if body.each completed.> In dealing with this I''m thinking about turning on > proxy_ignore_client_abort > (http://wiki.nginx.org/HttpProxyModule#proxy_ignore_client_abort) so > that requests that make it to the Rails > app aren''t aborted. Does anyone have experience with this? I can see it > causing its own sorts of confusion.I''ve never used it. -- Eric Wong
Jesse Storimer <jstorimer at gmail.com> wrote:> It seems that in this case the Rails app actually aborts the request, > wherever it is in the course of it. The issue I ran into is that my app > made a destructive request to an external service in > the context of a request, but the client disconnected before the app was > able to respond. So the external service returned its response but the > request was aborted before the app was able > to commit its transaction to the database, confusion ensued.You should make this request to the external service and wait for this response inside state 3) as described in my other reply[1]. If you''re affected by state 2), you should minimize the time in state 2) by using PrereadInput middleware to jump to state 3) (and you''ll avoid making the external request entirely if the preread input fails due to the client aborting). Your goal is to minimize the time spent processing a non-idempotent request/parts-of-the-request and isolate where a client can fail (and maximize the time where they can gracefully recover). If possible, the request you make to the external service should be idempotent, but it doesn''t seem to be... I haven''t encountered this problem myself in many years, but here''s what I did in the past for similar problems: Call the external service asynchronously and look at the various background job/queue libraries available to handle this. The client should probably auto-refresh on a page (idempotently) until the async request is complete. I would even hold off on starting your external request until the client has hit the auto-refresh page. This way you know the client has started the refresh cycle and you can fake the idempotency of the external request on your app side. [1] - http://mid.gmane.org/20110808192824.GA5759 at dcvr.yhbt.net -- Eric Wong
On Mon, Aug 08, 2011 at 07:32:52PM +0000, Eric Wong wrote:> (not sure if you''re subscribed or not, actually, I don''t see you in > mailman...) > > ----- Forwarded message from Eric Wong <normalperson at yhbt.net> ----- > > From: Eric Wong <normalperson at yhbt.net> > To: unicorn list <mongrel-unicorn at rubyforge.org> > Subject: Re: What happens when a client terminates a connection? > > Jesse Storimer <jstorimer at gmail.com> wrote: > > I''ve been trying to understand what happens in Unicorn when a client > > terminates a connection, and nginx logs a 499 response code. > > We''d have to read the nginx sources to answer what nginx does, but > of course I can answer what Unicorn does off the top of my head. > > > In my debugging this can happen if the client is on a flaky connection, > > or if they double-click a form submit button, the first request is > > terminated and nginx logs a 499 response code. > > <snip> (the snipped paragraph deserves independent observation/attention) > > > Can you confirm that this is actually what happens in Unicorn when the > > client disconnects? I''m not seeing anything in the logs to indicate the > > actual behaviour. > > It depends on when exactly the client (nginx) disconnect is detected. > Unicorn has 4 distinct states : > > 1) reading headers, if a client disconnects before it has written _all_ > of its request headers, the Rack app will never be called. > > Since no applicaton logic fired at this point. > > 2) inside Rack dispatch (rack.input reading) > This will abort the Rack application dispatch if your client > disconnects before _all_ of the request body is sent. Unlike > most servers, Unicorn lazily reads any request bodies. > > You can catch exceptions from env["rack.input"].{read,gets,each} > to detect this. > > The Unicorn::PrereadInput middleware can minimize the time window for > this state by reading the request body ASAP. > > You can also ignore this if your app isn''t handling > requests with bodies (POST/PUT), but since you mentioned form > input... > > 3) inside Rack dispatch (after rack.input reading) > Your app has no way of knowing your client disconnected at > this stage. You can hack Unicorn to IO.select in a separate > thread, but there''ll always be exposed windows leading up to > 4) so it''s not worth it... > > 4) writing the response: Unicorn will abort whenever a socket > error is detected. Keep in mind that every single part of the > Rack response array can be dynamically generated by the app. > Your application can still be "running" even though the Rack app > has returned its response for Unicorn to start writing. > > Clients/Rack middleware can be written to detect this in the > response body "close" method by checking if body.each completed. > > > In dealing with this I''m thinking about turning on > > proxy_ignore_client_abort > > (http://wiki.nginx.org/HttpProxyModule#proxy_ignore_client_abort) so > > that requests that make it to the Rails > > app aren''t aborted. Does anyone have experience with this? I can see it > > causing its own sorts of confusion. > > I''ve never used it. > > -- > Eric Wong > > ----- End forwarded message ----- > > -- > Eric WongThanks for that explanation. Just so I understand, once the Rack application enters 3) then it should be unaffected by a client disconnect, or any socket error? I''ll definitely give PrereadInput a try in that case. When you say that Unicorn lazily reads request bodies, do you mean that my Rails application might already be in the middle of processing the request but Unicorn is still reading from the client socket? At that point won''t Rails have read all of the request body? Or does that only apply if I stick a middleware at the front of the stack that does all the heavy lifting?
Jesse Storimer <jstorimer at gmail.com> wrote:> Eric Wong <normalperson at yhbt.net> wrote: > > Unicorn has 4 distinct states :<snip>> > 3) inside Rack dispatch (after rack.input reading) > > Your app has no way of knowing your client disconnected at > > this stage. You can hack Unicorn to IO.select in a separate > > thread, but there''ll always be exposed windows leading up to > > 4) so it''s not worth it... > > > > 4) writing the response: Unicorn will abort whenever a socket > > error is detected. Keep in mind that every single part of the > > Rack response array can be dynamically generated by the app. > > Your application can still be "running" even though the Rack app > > has returned its response for Unicorn to start writing. > > > > Clients/Rack middleware can be written to detect this in the > > response body "close" method by checking if body.each completed. > > Thanks for that explanation. Just so I understand, once the Rack application > enters 3) then it should be unaffected by a client disconnect, or any > socket error? I''ll definitely give PrereadInput a try in that case.Yes, however I don''t think I made it clear that 3) will /always/ transition to state 4). So you''ll be able to use body.close to detect a client write failure.> When you say that Unicorn lazily reads request bodies, do you mean that > my Rails application might already be in the middle of processing the > request but Unicorn is still reading from the client socket?Yes. We use http://unicorn.bogomips.org/Unicorn::TeeInput.html by default so it gives the Rack app a chance to reject a client if it sees something it doesnt like> At that point won''t Rails have read all of the request body? Or does > that only apply if I stick a middleware at the front of the stack that > does all the heavy lifting?I''m not certain about middlewares. Rails may attempt to read all input ASAP anyways depending on the request Content-Type/Encoding. Check the Rails/Rack sources for your versions of Rails/Rack to be sure. -- Eric Wong