Hey guys, I''m running a Rails 3 beta3 application using Ruby 1.9.2-preview3 and it has been running fairly well, however every now and then (usually when under high load), I see error messages like: 2010/06/04 14:34:29 [error] 1884#0: *8844 upstream prematurely closed connection while reading response header from upstream, client: 119.235.237.93, server: portal.crowdreel.com, request: "GET /post/18Vmy HTTP/1.1", upstream: "http://unix:/home/app/crowdreel-portal/tmp/sockets/unicorn.sock:/post/18Vmy", host: "crowdreel.com" ... a lot of these messages.. this is a pretty heavy duty server with lots of ram. I have a setup similar to what I read up on github''s blog as in: nginx -> unicorns via a socket upstream. My unicorn.rb file looks like: -------- # 16 workers and 1 master worker_processes 16 # Load the app into the master before forking workers for super-fast worker spawn times preload_app false # Restart any workers that haven''t responded in 30 seconds timeout 45 # Listen on a Unix data socket listen File.expand_path(File.dirname(__FILE__)+''/../tmp/sockets/unicorn.sock''), :backlog => 2048 ---------- I don''t preload the app because I start running into annoying issues where my workers use the same db connection, so instead I just have them load their own instance, its very fast to start up anyways. I''ve read everything I could on this issue and exhausted Google, which leaves me to think the only thing left is some incompatibilities with 1.9.2. So I was wondering, how can I get logs from unicorn''s workers/master to know what is happening to these workers, perhaps something is segfaulting? Thx! Regards, Peter
Peter Kieltyka <peter.kieltyka at nulayer.com> wrote:> Hey guys, > > I''m running a Rails 3 beta3 application using Ruby 1.9.2-preview3 and > it has been running fairly well, however every now and then (usually > when under high load), I see error messages like: > > 2010/06/04 14:34:29 [error] 1884#0: *8844 upstream prematurely closed > connection while reading response header from upstream, client: > 119.235.237.93, server: portal.crowdreel.com, request: "GET > /post/18Vmy HTTP/1.1", upstream: > "http://unix:/home/app/crowdreel-portal/tmp/sockets/unicorn.sock:/post/18Vmy", > host: "crowdreel.com" > > ... a lot of these messages.. this is a pretty heavy duty server with > lots of ram. > > I have a setup similar to what I read up on github''s blog as in: nginx > -> unicorns via a socket upstream. > > My unicorn.rb file looks like: > > -------- > > # 16 workers and 1 master > worker_processes 16 > > # Load the app into the master before forking workers for super-fast worker spawn times > preload_app false > > # Restart any workers that haven''t responded in 30 seconds > timeout 45 > > # Listen on a Unix data socket > listen File.expand_path(File.dirname(__FILE__)+''/../tmp/sockets/unicorn.sock''), :backlog => 2048 > > ----------> I don''t preload the app because I start running into annoying issues > where my workers use the same db connection, so instead I just have > them load their own instance, its very fast to start up anyways. > > I''ve read everything I could on this issue and exhausted Google, which > leaves me to think the only thing left is some incompatibilities with > 1.9.2. So I was wondering, how can I get logs from unicorn''s > workers/master to know what is happening to these workers, perhaps > something is segfaulting?Hi Peter, Set stderr_path in your unicorn config file so Unicorn can log its errors to a file. Otherwise Unicorn will send all errors to /dev/null when daemonized. I shall emphasize the importance of stderr_path if you''re using the default logger configuration in the documentation. Any chance the "/post/18Vmy" request is hitting your timeout? The only way you can know is if Unicorn logged its errors. You might also want to check your Rails log for exceptions, too, as some uncaught errors may show up there. -- Eric Wong
Eric Wong <normalperson at yhbt.net> wrote:> I shall emphasize the importance of stderr_path if you''re using the > default logger configuration in the documentation.I''ve just pushed the following out to unicorn.git and website. Comments/grammar/speling korections welcome as always.>From 2d5a4b075801493a85c6e8d2dbdf0c95002e046d Mon Sep 17 00:00:00 2001From: Eric Wong <normalperson at yhbt.net> Date: Fri, 4 Jun 2010 13:42:34 -0700 Subject: [PATCH] doc: emphasize the importance of stderr_path While second nature to myself, stderr_path may be an overlooked configuration parameter for some users. Also, add a minimal sample configuration file that is shorter and hopefully less intimidating to new users. --- examples/unicorn.conf.minimal.rb | 13 +++++++++++++ examples/unicorn.conf.rb | 12 +++++++++--- lib/unicorn/configurator.rb | 22 ++++++++++++++++++---- 3 files changed, 40 insertions(+), 7 deletions(-) create mode 100644 examples/unicorn.conf.minimal.rb diff --git a/examples/unicorn.conf.minimal.rb b/examples/unicorn.conf.minimal.rb new file mode 100644 index 0000000..2a47910 --- /dev/null +++ b/examples/unicorn.conf.minimal.rb @@ -0,0 +1,13 @@ +# Minimal sample configuration file for Unicorn (not Rack) when used +# with daemonization (unicorn -D) started in your working directory. +# +# See http://unicorn.bogomips.org/Unicorn/Configurator.html for complete +# documentation. +# See also http://unicorn.bogomips.org/examples/unicorn.conf.rb for +# a more verbose configuration using more features. + +listen 2007 # by default Unicorn listens on port 8080 +worker_processes 2 # this should be >= nr_cpus +pid "/path/to/app/shared/pids/unicorn.pid" +stderr_path "/path/to/app/shared/log/unicorn.log" +stdout_path "/path/to/app/shared/log/unicorn.log" diff --git a/examples/unicorn.conf.rb b/examples/unicorn.conf.rb index e209894..37c3e81 100644 --- a/examples/unicorn.conf.rb +++ b/examples/unicorn.conf.rb @@ -1,4 +1,9 @@ -# Sample configuration file for Unicorn (not Rack) +# Sample verbose configuration file for Unicorn (not Rack) +# +# This configuration file documents many features of Unicorn +# that may not be needed for some applications. See +# http://unicorn.bogomips.org/examples/unicorn.conf.minimal.rb +# for a much simpler configuration file. # # See http://unicorn.bogomips.org/Unicorn/Configurator.html for complete # documentation. @@ -22,8 +27,9 @@ timeout 30 # feel free to point this anywhere accessible on the filesystem pid "/path/to/app/shared/pids/unicorn.pid" -# some applications/frameworks log to stderr or stdout, so prevent -# them from going to /dev/null when daemonized here: +# By default, the Unicorn logger will write to stderr. +# Additionally, ome applications/frameworks log to stderr or stdout, +# so prevent them from going to /dev/null when daemonized here: stderr_path "/path/to/app/shared/log/unicorn.stderr.log" stdout_path "/path/to/app/shared/log/unicorn.stdout.log" diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb index 64a25e3..4fa745d 100644 --- a/lib/unicorn/configurator.rb +++ b/lib/unicorn/configurator.rb @@ -7,9 +7,11 @@ module Unicorn # Implements a simple DSL for configuring a Unicorn server. # - # See http://unicorn.bogomips.org/examples/unicorn.conf.rb for an - # example config file. An example config file for use with nginx is - # also available at http://unicorn.bogomips.org/examples/nginx.conf + # See http://unicorn.bogomips.org/examples/unicorn.conf.rb and + # http://unicorn.bogomips.org/examples/unicorn.conf.minimal.rb + # example configuration files. An example config file for use with + # nginx is also available at + # http://unicorn.bogomips.org/examples/nginx.conf class Configurator < Struct.new(:set, :config_file, :after_reload) # Default settings for Unicorn @@ -78,6 +80,10 @@ module Unicorn # sets object to the +new+ Logger-like object. The new logger-like # object must respond to the following methods: # +debug+, +info+, +warn+, +error+, +fatal+, +close+ + # The default Logger will log its output to the path specified + # by +stderr_path+. If you''re running Unicorn daemonized, then + # you must specify a path to prevent error messages from going + # to /dev/null. def logger(new) %w(debug info warn error fatal close).each do |m| new.respond_to?(m) and next @@ -310,11 +316,19 @@ module Unicorn # file will be opened with the File::APPEND flag and writes # synchronized to the kernel (but not necessarily to _disk_) so # multiple processes can safely append to it. + # + # If you are daemonizing and using the default +logger+, it is important + # to specify this as errors will otherwise be lost to /dev/null. + # Some applications/libraries may also triggering warnings that go to + # stderr, and they will end up here. def stderr_path(path) set_path(:stderr_path, path) end - # Same as stderr_path, except for $stdout + # Same as stderr_path, except for $stdout. Not many Rack applications + # write to $stdout, but any that do will have their output written here. + # It is safe to point this to the same location a stderr_path. + # Like stderr_path, this defaults to /dev/null when daemonized. def stdout_path(path) set_path(:stdout_path, path) end -- Eric Wong
Thanks Eric! It really helps knowing the stderr_path configuration option :) doh. I haven''t yet run into the issue since, but I will have my unicorn logs ready when it does. By the way, have you ever thought about moving the unicorn mailing list to a Google group? It''s much easier to interact with, search, communicate, etc. Just saying... Even issues like trying to post this reply here, and getting "The following errors were found: You have lines longer than 80 characters. Fix that." ... hehe, thats pretty funny. Cheers. Peter
Peter <peter.kieltyka at nulayer.com> wrote:> By the way, have you ever thought about moving the unicorn mailing list to > a Google group? It''s much easier to interact with, search, communicate, etc. > Just saying...Google Groups get more spam, and last I checked there''s no way to download mail archives easily other than scraping the site. The Rubyforge at least has that, but the searchability sucks (so we mirror to gmane/mail-archive). Librelist is nice, but didn''t exist when I started this project and at this point the pain of migration is more than the benefit. I''ll consider Librelist if there''s strong support on the mailing list here.> Even issues like trying to post this reply here, and getting "The following > errors were found: You have lines longer than 80 characters. Fix that." > ... hehe, thats pretty funny.Where does it say that? I usually wrap my replies at 72 characters, but there should be nothing in the mailman config that prevents longer lines from being posted (and some patches do go longer than 80 due to the +/- prefixes). Of course HTML email is hugely wasteful and that''s not allowed here. -- Eric Wong