thr3ads.net - mongrel unicorn - Exception `Errno::EAGAIN'' [Nov 2009]

If this information is useful, please help other people find it:
Share via:

James Turnbull

2009-Nov-15 23:21 UTC

Exception `Errno::EAGAIN''

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I get this repeated error - about once a second:

Exception `Errno::EAGAIN'' at
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
Resource temporarily unavailable - accept(2)
Exception `Errno::EAGAIN'' at
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
Resource temporarily unavailable - accept(2)
Exception `Errno::EAGAIN'' at
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
Resource temporarily unavailable - accept(2)

Which seems to be related to:

        ready.each do |sock|
          begin
            process_client(sock.accept_nonblock)
            nr += 1
            alive.chmod(m = 0 == m ? 1 : 0)
          rescue Errno::EAGAIN, Errno::ECONNABORTED
          end
          break if nr < 0
        end

My unicorn.conf is:

worker_processes 4
working_directory "/etc/puppet"
listen ''/tmp/puppet.sock'', :backlog => 1
listen 8140, :tcp_nopush => true
timeout 10
pid "/tmp/puppet.pid"

  preload_app true
  GC.respond_to?(:copy_on_write_friendly=) and
  GC.copy_on_write_friendly = true

  before_fork do |server, worker|
    # the following is recomended for Rails + "preload_app true"
    # as there''s no need for the master process to hold a connection
#    defined?(ActiveRecord::Base) and
#      ActiveRecord::Base.connection.disconnect!

    # the following allows a new master process to incrementally
    # phase out the old master process with SIGTTOU to avoid a
    # thundering herd (especially in the "preload_app false" case)
    # when doing a transparent upgrade.  The last worker spawned
    # will then kill off the old master process with a SIGQUIT.
    old_pid = "#{server.config[:pid]}.oldbin"
    if old_pid != server.pid
      begin
        sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
        Process.kill(sig, File.read(old_pid).to_i)
      rescue Errno::ENOENT, Errno::ESRCH
      end
    end

    # optionally throttle the master from forking too quickly by
sleeping
    sleep 1
  end

The platform is Fedora 10, Ruby 1.8.6.

Oddly, if I comment out the preload I get:

Exception `Errno::EPERM'' at
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:639 -
Operation not permitted - /tmp/0.520680132392046
Unhandled listen loop exception #<Errno::EPERM: Operation not
permitted - /tmp/0.520680132392046>.
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:639:in
`chmod''
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:639:in
`worker_loop''
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:534:in
`spawn_missing_workers''
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:534:in `fork''
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:534:in
`spawn_missing_workers''
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:530:in `each''
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:530:in
`spawn_missing_workers''
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:540:in
`maintain_worker_count''
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:215:in
`start''
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:28:in `run''
/usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/bin/unicorn:165
/usr/bin/unicorn:19:in `load''
/usr/bin/unicorn:19

Repeatedly.

Regards

James Turnbull

- --
Author of:
* Pro Linux System Administration (http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet (http://tinyurl.com/pupbook)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEVAwUBSwCM/iFa/lDkFHAyAQIiSggA5r3rFPGiY5AeratLgxPvUDE1OjnrhRGn
HxoRAcM901BNvXT5i7fX5fMTayW7vPXiKd0IazottCDgd40TYlFTQZNpU9hUDx+g
LRAKQ5M5gHTkFX6nqxFlys8WIA3xseDBBUBNxGnZ4YGbqUMg7tQ258b41hiU+4pW
fxQgzYN7JjRQ7/u1Px+86aEEdSGHFYNAOSwG15b+iE/7hQbiWrlFIL7p7XcZqT8M
UxDm6eTkhP1kry7zJ4wTxVsreXgCi75AaCJBfIIoKs0NMParJ/rBWB+42ZA+1t5/
aEaKXBOi47Kw3CmbsSh6Bi4zbTyuavCX6/tfOMNXx6jXygvykUzpgw==pJbE
-----END PGP SIGNATURE-----

Eric Wong

2009-Nov-15 23:47 UTC

head link

Exception `Errno::EAGAIN''

James Turnbull <james at lovedthanlost.net> wrote:> I get this repeated error - about once a second:
> 
> Exception `Errno::EAGAIN'' at
> /usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
> Resource temporarily unavailable - accept(2)
> Exception `Errno::EAGAIN'' at
> /usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
> Resource temporarily unavailable - accept(2)
> Exception `Errno::EAGAIN'' at
> /usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:643 -
> Resource temporarily unavailable - accept(2)
> 
> Which seems to be related to:
> 
>         ready.each do |sock|
>           begin
>             process_client(sock.accept_nonblock)
>             nr += 1
>             alive.chmod(m = 0 == m ? 1 : 0)
>           rescue Errno::EAGAIN, Errno::ECONNABORTED
>           end
>           break if nr < 0
>         end
Hi James,

This is expected.  The kernel wakes up all the workers when there''s
_one_ connection and they all race to accept() one client connection.
One wins and accepts the connection, 3 lose and go back to sleep.  With
lots of worker processes, this can be a thundering herd problem but
that''s why we encourage Unicorn as a backend server, not as a frontend
server.

If we only have one listener, we could do a blocking accept() call to
avoid thundering herds under Linux, but testing on a 16-core box, I
don''t remember being able to measure a performance improvement.  But
otherwise we have to use select() with multiple listeners.  nginx uses a
variety of non-portable locking mechanisms to implement its accept
mutex, but that''s too much work under Ruby for little/no benefit.  In
the nginx case, sometimes even disabling the accept mutex entirely gives
better performance.
> My unicorn.conf is:
> 
> worker_processes 4
> working_directory "/etc/puppet"
> listen ''/tmp/puppet.sock'', :backlog => 1
> listen 8140, :tcp_nopush => true
> timeout 10
> pid "/tmp/puppet.pid"
> 
>   preload_app true
>   GC.respond_to?(:copy_on_write_friendly=) and
>   GC.copy_on_write_friendly = true
> 
>   before_fork do |server, worker|
>     # the following is recomended for Rails + "preload_app true"
>     # as there''s no need for the master process to hold a
connection
> #    defined?(ActiveRecord::Base) and
> #      ActiveRecord::Base.connection.disconnect!
> 
>     # the following allows a new master process to incrementally
>     # phase out the old master process with SIGTTOU to avoid a
>     # thundering herd (especially in the "preload_app false"
case)
>     # when doing a transparent upgrade.  The last worker spawned
>     # will then kill off the old master process with a SIGQUIT.
>     old_pid = "#{server.config[:pid]}.oldbin"
>     if old_pid != server.pid
>       begin
>         sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
>         Process.kill(sig, File.read(old_pid).to_i)
>       rescue Errno::ENOENT, Errno::ESRCH
>       end
>     end
> 
>     # optionally throttle the master from forking too quickly by
> sleeping
>     sleep 1
>   end
> 
> The platform is Fedora 10, Ruby 1.8.6.
> 
> Oddly, if I comment out the preload I get:
> 
> Exception `Errno::EPERM'' at
> /usr/lib/ruby/gems/1.8/gems/unicorn-0.95.0/lib/unicorn.rb:639 -
> Operation not permitted - /tmp/0.520680132392046
Do you by any chance do user switching in your application
or config.ru?

Instead, since 0.94.0, I would do this:

	after_fork do |server, worker|
	  worker.user(''user'', ''group'') if
Process.euid == 0
	end

That way only the worker process drops permissions.  The master really
doesn''t do anything interesting.   Of course *I* would just start
Unicorn as a regular user and forget the complexity of user switching
entirely...

-- 
Eric Wong

James Turnbull

2009-Nov-16 00:08 UTC

head link

Exception `Errno::EAGAIN''

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Eric Wong wrote:> This is expected.  The kernel wakes up all the workers when
there''s
> _one_ connection and they all race to accept() one client connection.
> One wins and accepts the connection, 3 lose and go back to sleep.  With
> lots of worker processes, this can be a thundering herd problem but
> that''s why we encourage Unicorn as a backend server, not as a
frontend
> server.
Ah okay - makes sense.
> Do you by any chance do user switching in your application
> or config.ru?
In the application we do.
> Instead, since 0.94.0, I would do this:
> 
> 	after_fork do |server, worker|
> 	  worker.user(''user'', ''group'') if
Process.euid == 0
> 	end
> 
> That way only the worker process drops permissions.  The master really
> doesn''t do anything interesting.   Of course *I* would just start
> Unicorn as a regular user and forget the complexity of user switching
> entirely...
Okay thanks - will give that a shot.

Regards

James Turnbull

- --
Author of:
* Pro Linux System Administration (http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet (http://tinyurl.com/pupbook)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEVAwUBSwCYGCFa/lDkFHAyAQIyPggA5XPzSOYgf4ekdBGoRcs4wp885eC8C/wY
BIjzdxilCEhA8mqBw4Hrjo5sOebglOukACFla7QaDqyk2t1velQEpxj8sR2StNEe
HHTbng/Jq3ryt4BxNucQWdKvdPTI3WJ3I9APqtGxNyh72FpY08uvAlUv02S8b/Aj
mtFqbqi2gu+tkbOoqW1AeJEabjJHd7HyPhf2YlHowBaHG61bzrFRnuhvqN8R6SAq
j3LI1kRpnTYdwKYEy3sLfR0OUSVmUuDkWjRkYt2tR2fAwCKkq2oupmcDUrvaT2Y2
fbQ8YJzoGjwVVvD1On/u6iNiTHZcOReYiHRsAhmtbaS+eHldSukHQw==EyRA
-----END PGP SIGNATURE-----

mongrel unicorn - Nov 2009 - Exception `Errno::EAGAIN'

Exception `Errno::EAGAIN''

Exception `Errno::EAGAIN''

Exception `Errno::EAGAIN''