thr3ads.net - mongrel unicorn - Address already in use [Jun 2012]

If this information is useful, please help other people find it:
Share via:

Manuel Palenciano Guerrero

2012-Jun-25 13:02 UTC

Address already in use

Hello there,

I seem to have a problem with unix-sockets, and cannot see many people with the
same situation when googling.

The problem is when upgrading (USR2 + QUIT) our applications. I get the
following error very frequently but not always.

E, [2012-06-21T11:40:46.386486 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.app.sock (in use)
E, [2012-06-21T11:40:46.386669 #29401] ERROR -- : retrying in 0.5 seconds (4
tries left)
E, [2012-06-21T11:40:46.887724 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.app.sock (in use)
E, [2012-06-21T11:40:46.887832 #29401] ERROR -- : retrying in 0.5 seconds (3
tries left)
E, [2012-06-21T11:40:47.388813 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.app.sock (in use)
E, [2012-06-21T11:40:47.388894 #29401] ERROR -- : retrying in 0.5 seconds (2
tries left)
E, [2012-06-21T11:40:47.889878 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.app.sock (in use)
E, [2012-06-21T11:40:47.889957 #29401] ERROR -- : retrying in 0.5 seconds (1
tries left)
E, [2012-06-21T11:40:48.390939 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.app.sock (in use)
E, [2012-06-21T11:40:48.391020 #29401] ERROR -- : retrying in 0.5 seconds (0
tries left)
E, [2012-06-21T11:40:48.892002 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.app.sock (in use)
/var/www/app/staging/shared/bundle/ruby/1.8/gems/unicorn-4.3.0/lib/unicorn/socket_helper.rb:140:in
`initialize'': Address already in use - /tmp/unicorn.app.sock
(Errno::EADDRINUSE)

...and the only way around that I know is stoping and starting the app, even in
production.

We have preload_app => true, and ONLY listening on unix-sockets, no
TCP-sockets.

The only solution I can think of would we switching to TCP, but would there be a
reason on doing such ?

Is this happening to any body else ? and would you know a possible solution ?

Thanks very much in advance !

Manuel Palenciano

Jérémy Lecour

2012-Jun-25 13:41 UTC

head link

Address already in use

Le 25 juin 2012 ? 15:02, Manuel Palenciano Guerrero a ?crit :
> Hello there,
> 
> I seem to have a problem with unix-sockets, and cannot see many people with
the same situation when googling.
> 
> The problem is when upgrading (USR2 + QUIT) our applications. I get the
following error very frequently but not always.
> 
> E, [2012-06-21T11:40:46.386486 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.app.sock (in use)
> E, [2012-06-21T11:40:46.386669 #29401] ERROR -- : retrying in 0.5 seconds
(4 tries left)
> E, [2012-06-21T11:40:46.887724 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.app.sock (in use)
> E, [2012-06-21T11:40:46.887832 #29401] ERROR -- : retrying in 0.5 seconds
(3 tries left)
> E, [2012-06-21T11:40:47.388813 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.app.sock (in use)
> E, [2012-06-21T11:40:47.388894 #29401] ERROR -- : retrying in 0.5 seconds
(2 tries left)
> E, [2012-06-21T11:40:47.889878 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.app.sock (in use)
> E, [2012-06-21T11:40:47.889957 #29401] ERROR -- : retrying in 0.5 seconds
(1 tries left)
> E, [2012-06-21T11:40:48.390939 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.app.sock (in use)
> E, [2012-06-21T11:40:48.391020 #29401] ERROR -- : retrying in 0.5 seconds
(0 tries left)
> E, [2012-06-21T11:40:48.892002 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.app.sock (in use)
>
/var/www/app/staging/shared/bundle/ruby/1.8/gems/unicorn-4.3.0/lib/unicorn/socket_helper.rb:140:in
`initialize'': Address already in use - /tmp/unicorn.app.sock
(Errno::EADDRINUSE)
> 
> ...and the only way around that I know is stoping and starting the app,
even in production.
> 
> We have preload_app => true, and ONLY listening on unix-sockets, no
TCP-sockets.
> 
> The only solution I can think of would we switching to TCP, but would there
be a reason on doing such ?
> 
> Is this happening to any body else ? and would you know a possible solution
?

Hi,

I''ve had the same issue, in the exact same context, and I''ve
found found a fix.

As I was starting to play with Unicorn, I copied/pasted an init script and an
Unicorn config script separately.

They were both trying to do a rolling upgrade and obviously one of them failed.

I''ve remove the rolling upgrade from the init script and do it only in
the "before_fork" part of my Unicorn script.

You can find my configuration and init scripts in this Gist :
https://gist.github.com/2988648


I hope this helps you.


J?r?my Lecour
Conception et d?veloppement d''applications web
06 22 43 88 94 - http://jeremy.wordpress.com - http://twitter.com/jlecour

Aaron Suggs

2012-Jun-25 13:42 UTC

head link

Address already in use

On Monday, June 25, 2012 at 9:02 AM, Manuel Palenciano Guerrero
wrote:> The problem is when upgrading (USR2 + QUIT) our applications. I get the
following error very frequently but not always.
>  
> E, [2012-06-21T11:40:48.892002 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.app.sock (in use)
>
/var/www/app/staging/shared/bundle/ruby/1.8/gems/unicorn-4.3.0/lib/unicorn/socket_helper.rb:140:in
`initialize'': Address already in use - /tmp/unicorn.app.sock
(Errno::EADDRINUSE)
>  
> ...and the only way around that I know is stoping and starting the app,
even in production.
I''ve run in to this problem using the included example init.sh script.
The `sleep 2` on line 48 is too short in some cases.
(http://bogomips.org/unicorn.git/tree/examples/init.sh#n48)

I use a patched example init script that uses `ps` to monitor the upgrade
process instead of sleeping:

https://gist.github.com/2988633

It works well for me.

-Aaron

P.S. I''ve tried to contribute the patch upstream via this list, but it
keeps getting rejected for having the wrong content type. Then I gave up.
I''m fine chalking it up to my own stupidity/laziness?just wanted to put
it out there that contributing could be easier. :-)

Eric Wong

2012-Jun-25 18:10 UTC

head link

Address already in use

Aaron Suggs <aaron at ktheory.com> wrote:> P.S. I''ve tried to contribute the patch upstream via this list,
but it
> keeps getting rejected for having the wrong content type. Then I gave
> up. I''m fine chalking it up to my own stupidity/laziness?just
wanted
> to put it out there that contributing could be easier. :-)
I want it to be easy, too, but that includes being easy for me to
review.  Inline patches are easiest as I can quote relevant portions
of the email in the reply.

Did you try "git send-email"?  It helps you format messages correctly
if your MUA isn''t capable of that.

Otherwise, yes, you can upload the commit to any public git repository
and I can clone + run "git show" / "git log -p" to see your
changes and
reply to them inline.  It takes me longer and I''ll end up faking the
quoted portions (via git format-patch --stdout ... | sed -e ''s/^/>
''
when reviewing.

Eric Wong

2012-Jun-25 20:28 UTC

head link

Address already in use

Manuel Palenciano Guerrero <mpalenciano at gmail.com>
wrote:> Hello there,
> 
> I seem to have a problem with unix-sockets, and cannot see many people with
the same situation when googling.
> 
> The problem is when upgrading (USR2 + QUIT) our applications. I get the
following error very frequently but not always.
> 
> E, [2012-06-21T11:40:46.386486 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.app.sock (in use)
You should''ve seen an INFO message saying something like:

   inherited addr=/tmp/unicorn.app.sock fd=...

in your logs.

Can you share your unicorn config?  Are you using a before_exec hook
at all and could that hook be clobbering ENV["UNICORN_FD"]?

Manuel Palenciano Guerrero

2012-Jun-25 21:03 UTC

head link

Address already in use

Hi,

First, thanks Eric, J?r?my and Aaron for replying. I really appreciate it.

Yes Eric, I can see the line... "inherited addr=/tmp/unicorn.app.sock
fd=..."

here is the full log

-------------------------------------------------
I, [2012-06-21T11:40:44.282224 #29212]  INFO -- : inherited
addr=/tmp/unicorn.sublimma_staging.sock fd=3
I, [2012-06-21T11:40:44.282480 #29212]  INFO -- : Refreshing Gem list
master process ready
worker=0 ready
worker=1 ready
reaped #<Process::Status: pid=28683,exited(0)> worker=0
reaped #<Process::Status: pid=28684,exited(0)> worker=1
master complete
E, [2012-06-21T11:40:46.386486 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.sublimma_staging.sock (in use)
E, [2012-06-21T11:40:46.386669 #29401] ERROR -- : retrying in 0.5 seconds (4
tries left)
E, [2012-06-21T11:40:46.887724 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.sublimma_staging.sock (in use)
E, [2012-06-21T11:40:46.887832 #29401] ERROR -- : retrying in 0.5 seconds (3
tries left)
E, [2012-06-21T11:40:47.388813 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.sublimma_staging.sock (in use)
E, [2012-06-21T11:40:47.388894 #29401] ERROR -- : retrying in 0.5 seconds (2
tries left)
E, [2012-06-21T11:40:47.889878 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.sublimma_staging.sock (in use)
E, [2012-06-21T11:40:47.889957 #29401] ERROR -- : retrying in 0.5 seconds (1
tries left)
E, [2012-06-21T11:40:48.390939 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.sublimma_staging.sock (in use)
E, [2012-06-21T11:40:48.391020 #29401] ERROR -- : retrying in 0.5 seconds (0
tries left)
E, [2012-06-21T11:40:48.892002 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.sublimma_staging.sock (in use)
/var/www/sublimma/staging/shared/bundle/ruby/1.8/gems/unicorn-4.3.0/lib/unicorn/socket_helper.rb:140:in
`initialize'': Address already in use -
/tmp/unicorn.sublimma_staging.sock (Errno::EADDRINUSE)
-------------------------------------------------

my unicorn.rb: https://gist.github.com/2991110

and my production_init.sh: http://unicorn.bogomips.org/examples/init.sh

I was planning on trying the following...

adding the killing of the old_pid to the before_fork(), as in...

  old_pid = RAILS_ROOT + ''/tmp/pids/unicorn.pid.oldbin''
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end

and making the UPGRADE case in the "init.sh" to just...

--------------------------------------------------
upgrade)
	sig USR2 && echo "Upgraded" && exit 0
	echo >&2 "Couldn''t upgrade, starting
''$CMD'' instead"
	$CMD
	;;
--------------------------------------------------

Regards and thanks again !

Manuel P.

On Jun 25, 2012, at 10:28 PM, Eric Wong wrote:
> Manuel Palenciano Guerrero <mpalenciano at gmail.com> wrote:
>> Hello there,
>> 
>> I seem to have a problem with unix-sockets, and cannot see many people
with the same situation when googling.
>> 
>> The problem is when upgrading (USR2 + QUIT) our applications. I get the
following error very frequently but not always.
>> 
>> E, [2012-06-21T11:40:46.386486 #29401] ERROR -- : adding listener
failed addr=/tmp/unicorn.app.sock (in use)
> 
> You should''ve seen an INFO message saying something like:
> 
>   inherited addr=/tmp/unicorn.app.sock fd=...
> 
> in your logs.
> 
> Can you share your unicorn config?  Are you using a before_exec hook
> at all and could that hook be clobbering ENV["UNICORN_FD"]?
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn at rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying

Eric Wong

2012-Jun-25 21:10 UTC

head link

Address already in use

Aaron Suggs <aaron at ktheory.com> wrote:> I''ve run in to this problem using the included example init.sh
script.
> The `sleep 2` on line 48 is too short in some cases.
> (http://bogomips.org/unicorn.git/tree/examples/init.sh#n48)
> 
> I use a patched example init script that uses `ps` to monitor the
> upgrade process instead of sleeping:
> https://gist.github.com/2988633
(quoted from the README in the gist above)> The salient part of the current upgrade task is:
> 
>    sig USR2 && sleep 2 && sig 0 && oldsig QUIT
>    # then wait for the old pid to disappear
> 
> We found that on tiny or busy servers, the "sleep 2" was too
short. A
> too short sleep leaves the server in an undesirable state. The init
> script fails, and both the old and new unicorn processes are running.
> To resolve, I''d manually send a QUIT to the old master to clean
things
> up.
> 
> We tried increasing the sleep time, and this issue became less common,
> but still happens. I''d like to avoid the sleep altogether.
Yeah, I''m not too happy about the sleeps, either.
> Second, the old workers are terminated before the new workers are
> ready to handle requests. This causes requests to hang (for about 25
> seconds in our case) until the app is loaded and the new workers begin
> responding to requests. I''d like to minimize the impact that code
> deploys have on our app''s performance.
> 
> With this patch, the `upgrade` task:
> 
> 1. Sends a USR2 signal to the current (old) master. This seems to
> immediately rename the pid to pid.oldbin.
That''s good in your case.  Don''t rely on it being immediate on
a very
busy system.
> 2. Waits for the master pid to exist, and for that process to have
> children. When preload_app is true, the presence of child processes
> means that those workers are nearly ready to handle requests (afaik,
> they just need to execute the after_fork block)
Correct.  preload_app true makes it much faster to start large
workers up.
> 3. Once the master process has children, send a QUIT to the old
> master. (If the old master is already gone, that means the new master
> failed to start. Exit with an error. Otherwise, wait for the old
> master to spin down.)
> 
> Caveats: with my patch, it''s more likely that for a second both
old
> any new workers are responding to requests. We find that this
doesn''t
> usually happen, and it''s not bad if it does. The way I find child
> processes "ps --no-headers --ppid `cat $PID`" is a totally
linux-ism.
> If someone can suggest a more portable command (OS X, etc), I''d
> appreciate it.
I agree, I can''t accept the patch as-is with a ps(1) dependency like
that.
> (The first patch converts tabs to spaces, as is common for ruby projects).
NACK.  Don''t change the indentation style (of /any/ project) without
discussion and agreement of the project leaders first.

In this case, I''m strongly against this change.  The shell script is
not
Ruby.  Most of the init scripts in my systems use hard tabs for
indentation and I believe it''s friendlier to sysadmins (who may not be
familiar with Ruby at all).

Fwiw, my preferred C indentation style is hard tabs, but I continue to
use only 2 spaces in the unicorn_http parser C extension because the
code was inherited from Mongrel.  This also allows patches to flow
between projects more easily (unicorn <-> mongrel <-> thin <->
kcar) if
needed.
> I realize this patch may be particular to our use case and may not
> generally appropriate. I''d appreciate learning what scripts others
use
> for fast, reliable unicorn upgrades (capistrano-unicorn gem looks
> neat).
> 
> Thanks!
Anyways, thanks for the suggestions and sharing it here.  Perhaps
wrapping the non-portable ps(1) dependency with:

	case $(uname -o) in
	GNU/Linux)
		...
		;;
	*)
		...
		;;
	esac

would be beneficial given the amount of Linux users in production.

Eric Wong

2012-Jun-25 23:57 UTC

head link

Address already in use

Manuel Palenciano Guerrero <mpalenciano at gmail.com>
wrote:> Hi,
> 
> First, thanks Eric, J?r?my and Aaron for replying. I really appreciate it.
> 
> Yes Eric, I can see the line... "inherited addr=/tmp/unicorn.app.sock
fd=..."
> 
> here is the full log
> 
> -------------------------------------------------
> I, [2012-06-21T11:40:44.282224 #29212]  INFO -- : inherited
addr=/tmp/unicorn.sublimma_staging.sock fd=3
> I, [2012-06-21T11:40:44.282480 #29212]  INFO -- : Refreshing Gem list
> master process ready
> worker=0 ready
> worker=1 ready
> reaped #<Process::Status: pid=28683,exited(0)> worker=0
> reaped #<Process::Status: pid=28684,exited(0)> worker=1
> master complete
Ugh, lack of formatting caused by Rails mucking with Logger is annoying.
Can you add the following to your unicorn config?

  Configurator::DEFAULTS[:logger].formatter = Logger::Formatter.new

(that''s from unicorn.bogomips.org/FAQ.html)
> E, [2012-06-21T11:40:46.386486 #29401] ERROR -- : adding listener failed
addr=/tmp/unicorn.sublimma_staging.sock (in use)
I''m curious if PID=29401 is a worker of pid 29212.  Or you''re
starting
another master somewhere...
>
/var/www/sublimma/staging/shared/bundle/ruby/1.8/gems/unicorn-4.3.0/lib/unicorn/socket_helper.rb:140:in
`initialize'': Address already in use -
/tmp/unicorn.sublimma_staging.sock (Errno::EADDRINUSE)
> -------------------------------------------------
> 
> my unicorn.rb: https://gist.github.com/2991110
Did you share the correct config?  Your config has:

  listen "/tmp/unicorn.app_production.sock"

But your config has: /tmp/unicorn.sublimma_staging.sock

mongrel unicorn - Jun 2012 - Address already in use

Address already in use

Address already in use

Address already in use

Address already in use

Address already in use

Address already in use

Address already in use

Address already in use