Hi, This morning, while checking for a correct deployment, we found out that the Unicorns we are using were sending 500 Internal errors very frequently to the HAProxy that sits in front of them. After some investigation, It turned out that HAProxy checks the backend by opening and closing a connection to the unicorn. Unfortunately the Unicorns we use ( v 0.990.0 ) will try to reply to this probe by a 500, resulting in the error being sent to the HAProxy when the proxy has already close the connection. I think Unicorn should refrain from sending an error in this case. There is no visible side effect, excepted that there are a "lot" of 500s running on the wire that nobody will ever read. Also, these errors are not sent in the Unicorn log, making it quite difficult to understand what''s going on :-) Cheers, -- Pierre <oct at fotopedia.com> Server Shepherd at http://www.fotopedia.com/
Pierre, Any specific reason you''re using HAProxy for Unicorn? Clifton On Wed, Dec 1, 2010 at 5:59 AM, Pierre <oct at fotopedia.com> wrote:> Hi, > > This morning, while checking for a correct deployment, we found out > that the Unicorns we are using were sending 500 Internal errors very > frequently to the HAProxy that sits in front of them. After some > investigation, It turned out that HAProxy checks the backend by > opening and closing a connection to the unicorn. Unfortunately the > Unicorns we use ( v 0.990.0 ) will try to reply to this probe by a > 500, resulting in the error being sent to the HAProxy when the proxy > has already close the connection. I think Unicorn should refrain from > sending an error in this case. There is no visible side effect, > excepted that there are a "lot" of 500s running on the wire that > nobody will ever read. Also, these errors are not sent in the Unicorn > log, making it quite difficult to understand what''s going on :-) > > Cheers, > -- > Pierre <oct at fotopedia.com> > Server Shepherd at http://www.fotopedia.com/ > _______________________________________________ > Unicorn mailing list - mongrel-unicorn at rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-unicorn > Do not quote signatures (like this one) or top post when replying >
Pierre <oct at fotopedia.com> wrote:> Hi, > > This morning, while checking for a correct deployment, we found out > that the Unicorns we are using were sending 500 Internal errors very > frequently to the HAProxy that sits in front of them. After some > investigation, It turned out that HAProxy checks the backend by > opening and closing a connection to the unicorn. Unfortunately the > Unicorns we use ( v 0.990.0 ) will try to reply to this probe by a > 500, resulting in the error being sent to the HAProxy when the proxy > has already close the connection.Hi Pierre, HAProxy should be configured to send proper HTTP checks and not just TCP connection checks, the problem will go away then. Also, I can not recommend HAProxy unless you''re certain all your clients are on a LAN and can be trusted to never trickle uploads nor reading large responses. -- Eric Wong
Hi,> > Any specific reason you''re using HAProxy for Unicorn?Indeed, we have two front load-balancers instances running HAProxy and several backends running Unicorns. We use HAProxy to amortize part of the load and load-balance / failover between all the backends. We especially use the leastconn feature of HAProxy that ensure that the less powerful backends are queried less often. As we run on EC2 virtual machines, this feature has the please to help us maintaining a good QoS. Cheers, -- Pierre <oct at fotopedia.com> Server Shepherd at http://www.fotopedia.com/
Dear Eric, On Wed, Dec 1, 2010 at 5:52 PM, Eric Wong <normalperson at yhbt.net> wrote:> Hi Pierre, HAProxy should be configured to send proper HTTP checks and > not just TCP connection checks, the problem will go away then.I understood this could be fixed this way and we will probably do that soon. However, I think this is also the responsibility of Unicorn not to reply anything when there''s no request or at least log the error somewhere :)> Also, I > can not recommend HAProxy unless you''re certain all your clients are on > a LAN and can be trusted to never trickle uploads nor reading large > responses.While I understand that uploads are very complicated to handle on the stack (even nginx can be confused at upload sometimes), HAProxy proved it was very good at managing tons of connections and high volume traffic from the Internet. All the more so as it allows a very high level of redundancy at a very small cost that cannot be achieved simply otherwise. Do you have any pointers about your worrying non-recommendation of HAProxy ? As far as I''m concerned, I''ve used HAProxy in worldwide web context of website serving up to 25 Millions pages with no problem, ever. Thanks again for your support, :) -- Pierre <oct at fotopedia.com> Server Shepherd at http://www.fotopedia.com/
Pierre <oct at fotopedia.com> wrote:> On Wed, Dec 1, 2010 at 5:52 PM, Eric Wong <normalperson at yhbt.net> wrote: > > Hi Pierre, HAProxy should be configured to send proper HTTP checks and > > not just TCP connection checks, the problem will go away then. > > I understood this could be fixed this way and we will probably do that > soon. However, I think this is also the responsibility of Unicorn not > to reply anything when there''s no request or at least log the error > somewhere :)I''m not sure how Unicorn is actually replying to anything, does HAProxy write *anything* to the socket? Logging those bad connections is another DoS vector I''d rather avoid, and for connections where nothing is written, not even possible... If you have the TCP_DEFER_ACCEPT (Linux) or accept filters (FreeBSD), it''s highly likely the Unicorn would never see the connection if the client never wrote to it.> > Also, I > > can not recommend HAProxy unless you''re certain all your clients are on > > a LAN and can be trusted to never trickle uploads nor reading large > > responses. > > While I understand that uploads are very complicated to handle on the > stack (even nginx can be confused at upload sometimes), HAProxy proved > it was very good at managing tons of connections and high volume > traffic from the Internet. All the more so as it allows a very high > level of redundancy at a very small cost that cannot be achieved > simply otherwise. Do you have any pointers about your worrying > non-recommendation of HAProxy ?HAProxy starts writing request bodies to Unicorn as soon as the upload starts, which means the Unicorn process will be bounded by the speed of the original client connection. If multiple clients upload slowly, then you''ll end up hogging many Unicorn workers. nginx can also limit the size of client uploads (default 1M) to prevent unnecessary I/O. If you serve large responses that can''t fit in kernel socket buffers, then Unicorn will get stuck writing out to a client that isn''t reading fast enough. AFAIK, HAProxy also does not yet maintain keep-alive connections to clients, whereas nginx does. Keep-alive is important to client browsers, they can halve their active connections to a site if keep-alive is supported. -- Eric Wong
> HAProxy starts writing request bodies to Unicorn as soon as the upload > starts,Is that also true when the nginx upload module is used? (assuming nginx sits between HAProxy and the unicorns, otherwise it doesn''t make sense anyways). Unicorn is only hit after the complete upload is finished by nginx. I would assume this wouldn''t be different if HAProxy is put in front of all the nginx servers.> AFAIK, HAProxy also does not yet maintain keep-alive connections to > clients, whereas nginx does. Keep-alive is important to client browsers, > they can halve their active connections to a site if keep-alive is > supported.keep-alive options are in HAProxy since 1.4 There is further new development on this front if you''re using SSL. See the first quick news section at http://haproxy.1wt.eu/ : "The main advantage over the x-forwarded-for patch is that it now supports keep-alive" Lawrence
Lawrence Pit <lawrence.pit at gmail.com> wrote:> Eric Wong <normalperson at yhbt.net> wrote: >> HAProxy starts writing request bodies to Unicorn as soon as the upload >> starts, > > Is that also true when the nginx upload module is used? (assuming nginx > sits between HAProxy and the unicorns, otherwise it doesn''t make sense > anyways). Unicorn is only hit after the complete upload is finished by > nginx. I would assume this wouldn''t be different if HAProxy is put in > front of all the nginx servers.I would setup a test and confirm. You can do this trivially with the following Rack app: use Rack::ContentLength use Rack::ContentType run lambda { |env| input = env[''rack.input''] buf = input.each do |line| # each line is printed as Unicorn receives it, # if Unicorn is behind nginx, the times for each # line should be roughly/exactly the same. puts "#{Time.now} #{line.inspect}" end [ 200, {}, [] ] } And then telnetting or netcatting to your frontend, and typing the following (no leading spaces, use a proper hostname for vhosts): PUT / HTTP/1.1 Host: example.com Content-Length: 6 ...and then hit Enter a few times while watching your Unicorn terminal/logs. If you delay between hitting the Enter nginx should be buffering that.> keep-alive options are in HAProxy since 1.4 > > There is further new development on this front if you''re using SSL. See > the first quick news section at http://haproxy.1wt.eu/ : > > "The main advantage over the x-forwarded-for patch is that it now > supports keep-alive"Cool. -- Eric Wong
Hello, On Wed, Dec 1, 2010 at 8:58 PM, Eric Wong <normalperson at yhbt.net> wrote:> Pierre <oct at fotopedia.com> wrote: >> On Wed, Dec 1, 2010 at 5:52 PM, Eric Wong <normalperson at yhbt.net> wrote: >> > Hi Pierre, HAProxy should be configured to send proper HTTP checks and >> > not just TCP connection checks, the problem will go away then. >> >> I understood this could be fixed this way and we will probably do that >> soon. However, I think this is also the responsibility of Unicorn not >> to reply anything when there''s no request or at least log the error >> somewhere :) > > I''m not sure how Unicorn is actually replying to anything, does HAProxy > write *anything* to the socket? >It does. To reproduce the behavior, start a unicorn, start a sniffing tool (ngrep for example), connect to the unicorn and disconnect immediately: Connecting endpoint: [08:22][virtual] root at infrabox:~# telnet localhost 2002 Trying 127.0.0.1... Connected to infrabox.virtual.ftnz.net. Escape character is ''^]''. ^] telnet> quit Connection closed. Sniffing Console: [08:22][virtual] root at infrabox:~# ngrep "" port 2002 -d lo interface: lo (127.0.0.0/255.0.0.0) filter: (ip or ip6) and ( port 2002 ) ##### T 127.0.0.1:2002 -> 127.0.0.1:42874 [AFP] HTTP/1.1 500 Internal Server Error.... # Further packet inspection reveals that the 500 Internal Server Error is sent after the telnet has sent a FIN packet which means it will just be discarded. I understand logging it could be a DoS vector, but you must admit that seing 500''s from the server without any obvious reason can be frightening at first. Re HAProxy, and to complete Laurence reply from my point of view,> If you serve large responses that can''t fit in kernel socket buffers, > then Unicorn will get stuck writing out to a client that isn''t reading > fast enough.That can definitely be a problem in that case. To work around that we use a cache in the stack above the lower level of HAProxy (we have two other levels of HAProxy in the stack). Cheers, -- Pierre <oct at fotopedia.com> Server Shepherd at http://www.fotopedia.com/
Pierre <oct at fotopedia.com> wrote:> On Wed, Dec 1, 2010 at 8:58 PM, Eric Wong <normalperson at yhbt.net> wrote: > > I''m not sure how Unicorn is actually replying to anything, does HAProxy > > write *anything* to the socket? > > It does. To reproduce the behavior, start a unicorn, start a sniffing > tool (ngrep for example), connect to the unicorn and disconnect > immediately: > > Connecting endpoint: > > [08:22][virtual] root at infrabox:~# telnet localhost 2002 > Trying 127.0.0.1... > Connected to infrabox.virtual.ftnz.net. > Escape character is ''^]''. > ^] > telnet> quit > Connection closed.Hi, you started writing an HTTP message ("quit" can be a valid HTTP method name) and disconnected; don''t do that.> Further packet inspection reveals that the 500 Internal Server Error > is sent after the telnet has sent a FIN packet which means it will > just be discarded. I understand logging it could be a DoS vector, but > you must admit that seing 500''s from the server without any obvious > reason can be frightening at first.You can enable Ruby debugging (pass ''-d'' to unicorn) to see exactly which exception you got. Really, it shouldn''t matter, you should be sending real HTTP requests for health checks. -- Eric Wong
Pierre <oct at fotopedia.com> wrote:> On Wed, Dec 1, 2010 at 5:52 PM, Eric Wong <normalperson at yhbt.net> wrote: > > Also, I > > can not recommend HAProxy unless you''re certain all your clients are on > > a LAN and can be trusted to never trickle uploads nor reading large > > responses. > > While I understand that uploads are very complicated to handle on the > stack (even nginx can be confused at upload sometimes), HAProxy proved > it was very good at managing tons of connections and high volume > traffic from the Internet. All the more so as it allows a very high > level of redundancy at a very small cost that cannot be achieved > simply otherwise. Do you have any pointers about your worrying > non-recommendation of HAProxy ?On a related note, LWN has a new article[1] explaining the large body POST problem (more than a year since Slowloris, and ~3 years after the largely-unknown david[2]). [1] http://lwn.net/Articles/418017/ (subscribers) http://lwn.net/SubscriberLink/418017/1e63e408f34b5d68/ (non-subscribers) [2] http://git.bogomips.org/cgit/david.git -- Eric Wong
Hello, On Thu, Dec 2, 2010 at 6:39 PM, Eric Wong <normalperson at yhbt.net> wrote:>> >> Connecting endpoint: >> >> [08:22][virtual] root at infrabox:~# telnet localhost 2002 >> Trying 127.0.0.1... >> Connected to infrabox.virtual.ftnz.net. >> Escape character is ''^]''. >> ^] >> telnet> quit >> Connection closed. > > Hi, you started writing an HTTP message ("quit" can be a valid HTTP > method name) and disconnected; don''t do that. >No, quit is typed in the telnet console and not in the network stream. When passing -d to unicorn, I get the exception, that''s cool Exception `EOFError'' at /usr/local/rvm/gems/ree-1.8.7-2010.02/gems/unicorn-0.990.0/lib/unicorn/http_request.rb:59 - end of file reached Cheers, -- Pierre <oct at fotopedia.com> Server Shepherd at http://www.fotopedia.com/
Pierre <oct at fotopedia.com> wrote:> On Thu, Dec 2, 2010 at 6:39 PM, Eric Wong <normalperson at yhbt.net> wrote: > > Pierre <oct at fotopedia.com> wrote: > >> Connecting endpoint: > >> > >> [08:22][virtual] root at infrabox:~# telnet localhost 2002 > >> Trying 127.0.0.1... > >> Connected to infrabox.virtual.ftnz.net. > >> Escape character is ''^]''. > >> ^] > >> telnet> quit > >> Connection closed. > > > > Hi, you started writing an HTTP message ("quit" can be a valid HTTP > > method name) and disconnected; don''t do that. > > No, quit is typed in the telnet console and not in the network stream.Are you sure? With my telnet, "quit" definitely goes out to the server since you''re already connected. -- Eric Wong
russell muetzelfeldt
2010-Dec-05 01:04 UTC
Unicorn and HAProxy, 500 Internal errors after checks
On 05/12/2010, at 10:38 AM, Eric Wong wrote:> Pierre <oct at fotopedia.com> wrote: >> On Thu, Dec 2, 2010 at 6:39 PM, Eric Wong <normalperson at yhbt.net> wrote: >>> Pierre <oct at fotopedia.com> wrote: >>>> Connecting endpoint: >>>> >>>> [08:22][virtual] root at infrabox:~# telnet localhost 2002 >>>> Trying 127.0.0.1... >>>> Connected to infrabox.virtual.ftnz.net. >>>> Escape character is ''^]''. >>>> ^] >>>> telnet> quit >>>> Connection closed. >>> >>> Hi, you started writing an HTTP message ("quit" can be a valid HTTP >>> method name) and disconnected; don''t do that. >> >> No, quit is typed in the telnet console and not in the network stream. > > Are you sure? With my telnet, "quit" definitely goes out to the server > since you''re already connected.he connects, then sends an escape ("^]") to the telnet process, then types "quit" at the prompt the telnet process shows to tell the local telnet process to close the connection. the connection *might* see some traffic in attempted telnet option negotiation (unless the server end initiates that - I forget), but the "quit" string isn''t transmitted to the remote end. cheers Russell
russell muetzelfeldt <russm-rubyforge at slofith.org> wrote:> On 05/12/2010, at 10:38 AM, Eric Wong wrote: > > Pierre <oct at fotopedia.com> wrote: > >> On Thu, Dec 2, 2010 at 6:39 PM, Eric Wong <normalperson at yhbt.net> wrote: > >>> Pierre <oct at fotopedia.com> wrote: > >>>> Connecting endpoint: > >>>> > >>>> [08:22][virtual] root at infrabox:~# telnet localhost 2002 > >>>> Trying 127.0.0.1... > >>>> Connected to infrabox.virtual.ftnz.net. > >>>> Escape character is ''^]''. > >>>> ^] > >>>> telnet> quit > >>>> Connection closed. > >>> > >>> Hi, you started writing an HTTP message ("quit" can be a valid HTTP > >>> method name) and disconnected; don''t do that. > >> > >> No, quit is typed in the telnet console and not in the network stream. > > > > Are you sure? With my telnet, "quit" definitely goes out to the server > > since you''re already connected.> he connects, then sends an escape ("^]") to the telnet process, then > types "quit" at the prompt the telnet process shows to tell the local > telnet process to close the connection.Ah, I missed the "^]" :x> the connection *might* see some traffic in attempted telnet option > negotiation (unless the server end initiates that - I forget), but the > "quit" string isn''t transmitted to the remote end.Due to TCP_DEFER_ACCEPT, older Linux kernels won''t even wake up Unicorn. Newer ones treat the TCP_DEFER_ARGUMENT as a timeout but it''s just a quick EOFError. -- Eric Wong