I found this thread http://rubyforge.org/pipermail/mongrel-unicorn/2009-September/000031.html after googling for the same error located in my nginx error log. Is there some way to discover the current queue length of the backlog? Increasing the maximum size to 2048 "worked" for me as well, but obviously clients sit in the queue for a long time. If I could monitor queue length I could get some idea of when I have insufficient unicorn workers before the delay gets very high or requests fail. Thanks in advance! -Greg
ghazel at gmail.com wrote:> I found this thread > http://rubyforge.org/pipermail/mongrel-unicorn/2009-September/000031.html > after googling for the same error located in my nginx error log. > > Is there some way to discover the current queue length of the backlog? > Increasing the maximum size to 2048 "worked" for me as well, but > obviously clients sit in the queue for a long time. If I could monitor > queue length I could get some idea of when I have insufficient unicorn > workers before the delay gets very high or requests fail.If you''re using Linux, you can use Raindrops[1] to interrogate via inet_diag or parse /proc/net/{tcp,unix} yourself. You can run Raindrops separate/standalone script: http://git.bogomips.org/cgit/raindrops.git/tree/examples/linux-tcp-listener-stats.rb Or use it as middleware (see webpage): http://raindrops.bogomips.org/ git clone git://git.bogomips.org/raindrops -- Eric Wong
On Thu, Dec 2, 2010 at 6:39 PM, Eric Wong <normalperson at yhbt.net> wrote:> ghazel at gmail.com wrote: >> I found this thread >> http://rubyforge.org/pipermail/mongrel-unicorn/2009-September/000031.html >> after googling for the same error located in my nginx error log. >> >> Is there some way to discover the current queue length of the backlog? >> Increasing the maximum size to 2048 "worked" for me as well, but >> obviously clients sit in the queue for a long time. If I could monitor >> queue length I could get some idea of when I have insufficient unicorn >> workers before the delay gets very high or requests fail. > > If you''re using Linux, you can use Raindrops[1] to interrogate > via inet_diag or parse /proc/net/{tcp,unix} yourself. > > You can run Raindrops separate/standalone script: > http://git.bogomips.org/cgit/raindrops.git/tree/examples/linux-tcp-listener-stats.rb > > Or use it as middleware (see webpage): > http://raindrops.bogomips.org/ > > git clone git://git.bogomips.org/raindropsAmazing! I had somehow never heard of it. I use a unix socket for unicorn (should I not?) so I had to make a copy of linux-tcp-listener-stats.rb to support that. Now I''m able to get stats for my unicorn master. However, there seems to be some problem getting stats on TCP sockets: $ ruby linux-tcp-listener-stats.rb 127.0.0.1:9000 address active queued linux-tcp-listener-stats.rb:42:in `tcp_listener_stats'': Connection refused - sendmsg (Errno::ECONNREFUSED) from linux-tcp-listener-stats.rb:42 $ grep 0100007F:2328 /proc/net/tcp 1: 0100007F:2328 00000000:0000 0A 00000000:00000000 00:00000000 00000000 500 0 25307310 1 ffff880134471700 3000 0 0 2 -1 None of the TCP sockets I know to exist seem to work - they all fail with that error. Any ideas? -Greg
ghazel at gmail.com wrote:> On Thu, Dec 2, 2010 at 6:39 PM, Eric Wong <normalperson at yhbt.net> wrote: > > ghazel at gmail.com wrote: > >> I found this thread > >> http://rubyforge.org/pipermail/mongrel-unicorn/2009-September/000031.html > >> after googling for the same error located in my nginx error log. > >> > >> Is there some way to discover the current queue length of the backlog? > >> Increasing the maximum size to 2048 "worked" for me as well, but > >> obviously clients sit in the queue for a long time. If I could monitor > >> queue length I could get some idea of when I have insufficient unicorn > >> workers before the delay gets very high or requests fail. > > > > If you''re using Linux, you can use Raindrops[1] to interrogate > > via inet_diag or parse /proc/net/{tcp,unix} yourself. > > > > You can run Raindrops separate/standalone script: > > http://git.bogomips.org/cgit/raindrops.git/tree/examples/linux-tcp-listener-stats.rb > > > > Or use it as middleware (see webpage): > > http://raindrops.bogomips.org/ > > > > git clone git://git.bogomips.org/raindrops > > Amazing! I had somehow never heard of it.Yeah, I don''t have much interest in marketing; that''s what users are for, but it''s a chicken-and-egg problem :x> I use a unix socket for unicorn (should I not?)UNIX sockets are faster for nginx <-> Unicorn, but Raindrops is slower dealing with them since Linux doesn''t export UNIX sockets stats to netlink the same way it does for TCP sockets. So there''s always a tradeoff :x> so I had to make a copy of > linux-tcp-listener-stats.rb to support that. Now I''m able to get stats > for my unicorn master > > However, there seems to be some problem getting stats on TCP sockets: > > $ ruby linux-tcp-listener-stats.rb 127.0.0.1:9000 > address active queued > linux-tcp-listener-stats.rb:42:in `tcp_listener_stats'': Connection > refused - sendmsg (Errno::ECONNREFUSED) > from linux-tcp-listener-stats.rb:42 > $ grep 0100007F:2328 /proc/net/tcp > 1: 0100007F:2328 00000000:0000 0A 00000000:00000000 00:00000000 > 00000000 500 0 25307310 1 ffff880134471700 3000 0 0 2 -1 > > None of the TCP sockets I know to exist seem to work - they all fail > with that error.Odd, which kernel do you have? You might want to "modprobe inet_diag" if you''re on an older kernel. Or you can just comment-out the TCP code and use the UNIX sockets code. -- Eric Wong
On Thu, Dec 2, 2010 at 11:35 PM, Eric Wong <normalperson at yhbt.net> wrote:> ghazel at gmail.com wrote: >> On Thu, Dec 2, 2010 at 6:39 PM, Eric Wong <normalperson at yhbt.net> wrote: >> > ghazel at gmail.com wrote: >> >> I found this thread >> >> http://rubyforge.org/pipermail/mongrel-unicorn/2009-September/000031.html >> >> after googling for the same error located in my nginx error log. >> >> >> >> Is there some way to discover the current queue length of the backlog? >> >> Increasing the maximum size to 2048 "worked" for me as well, but >> >> obviously clients sit in the queue for a long time. If I could monitor >> >> queue length I could get some idea of when I have insufficient unicorn >> >> workers before the delay gets very high or requests fail. >> > >> > If you''re using Linux, you can use Raindrops[1] to interrogate >> > via inet_diag or parse /proc/net/{tcp,unix} yourself. >> > >> > You can run Raindrops separate/standalone script: >> > http://git.bogomips.org/cgit/raindrops.git/tree/examples/linux-tcp-listener-stats.rb >> > >> > Or use it as middleware (see webpage): >> > http://raindrops.bogomips.org/ >> > >> > git clone git://git.bogomips.org/raindrops >> >> Amazing! I had somehow never heard of it. > > Yeah, I don''t have much interest in marketing; that''s what users are > for, but it''s a chicken-and-egg problem :x > >> I use a unix socket for unicorn (should I not?) > > UNIX sockets are faster for nginx <-> Unicorn, but Raindrops is slower > dealing with them since Linux doesn''t export UNIX sockets stats to > netlink the same way it does for TCP sockets. ?So there''s always a > tradeoff :xWhen does that netlink speed difference matter - only when collecting stats?>> so I had to make a copy of >> linux-tcp-listener-stats.rb to support that. Now I''m able to get stats >> for my unicorn master >> >> However, there seems to be some problem getting stats on TCP sockets: >> >> $ ruby linux-tcp-listener-stats.rb 127.0.0.1:9000 >> ? ? ? ? ? ? address ? ? active ? ? queued >> linux-tcp-listener-stats.rb:42:in `tcp_listener_stats'': Connection >> refused - sendmsg (Errno::ECONNREFUSED) >> ? ? ? ? from linux-tcp-listener-stats.rb:42 >> $ grep ?0100007F:2328 /proc/net/tcp >> ? ?1: 0100007F:2328 00000000:0000 0A 00000000:00000000 00:00000000 >> 00000000 ? 500 ? ? ? ?0 25307310 1 ffff880134471700 3000 0 0 2 -1 >> >> None of the TCP sockets I know to exist seem to work - they all fail >> with that error. > > Odd, which kernel do you have? ?You might want to "modprobe inet_diag" > if you''re on an older kernel. ?Or you can just comment-out the TCP > code and use the UNIX sockets code.Linux hostname 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 x86_64 x86_64 GNU/Linux Using "modprobe inet_diag" made the script spit out a new error: address active queued linux-tcp-listener-stats.rb:42:in `tcp_listener_stats'': NLMSG_ERROR (RuntimeError) from linux-tcp-listener-stats.rb:42 $ sudo modprobe -l inet_diag /lib/modules/2.6.21.7-2.fc8xen/kernel/net/ipv4/inet_diag.ko I dug in to raindrops a little bit, and it seems the error associated with the NLMSG_ERROR is -2, "No such file or directory". -Greg
ghazel at gmail.com wrote:> On Thu, Dec 2, 2010 at 11:35 PM, Eric Wong <normalperson at yhbt.net> wrote: > > UNIX sockets are faster for nginx <-> Unicorn, but Raindrops is slower > > dealing with them since Linux doesn''t export UNIX sockets stats to > > netlink the same way it does for TCP sockets. ?So there''s always a > > tradeoff :x > > When does that netlink speed difference matter - only when collecting stats?Yes, the performance difference is only when collecting stats.> Using "modprobe inet_diag" made the script spit out a new error: > > address active queued > linux-tcp-listener-stats.rb:42:in `tcp_listener_stats'': NLMSG_ERROR > (RuntimeError) > from linux-tcp-listener-stats.rb:42I don''t think I''ve ever seen NLMSG_ERROR before. Are you running this as the same user that originally bound the listener (the user of the Unicorn master process)?> $ sudo modprobe -l inet_diag > /lib/modules/2.6.21.7-2.fc8xen/kernel/net/ipv4/inet_diag.ko > > I dug in to raindrops a little bit, and it seems the error associated > with the NLMSG_ERROR is -2, > "No such file or directory".Are you bound to 0.0.0.0:9000 and not 127.0.0.1:9000? Use 0.0.0.0:9000 if you''re bound to that (I had another user on the raindrops mailing list with that problem). -- Eric Wong
On Fri, Dec 3, 2010 at 1:49 PM, Eric Wong <normalperson at yhbt.net> wrote:> ghazel at gmail.com wrote: >> Using "modprobe inet_diag" made the script spit out a new error: >> >> ? ? ? ? ? ? address ? ? active ? ? queued >> linux-tcp-listener-stats.rb:42:in `tcp_listener_stats'': NLMSG_ERROR >> (RuntimeError) >> ? ? ? ? from linux-tcp-listener-stats.rb:42 > > I don''t think I''ve ever seen NLMSG_ERROR before. ?Are you running > this as the same user that originally bound the listener (the user > of the Unicorn master process)?Yes. I also tried as root, same error.>> $ sudo modprobe -l inet_diag >> /lib/modules/2.6.21.7-2.fc8xen/kernel/net/ipv4/inet_diag.ko >> >> I dug in to raindrops a little bit, and it seems the error associated >> with the NLMSG_ERROR is -2, >> "No such file or directory". > > Are you bound to 0.0.0.0:9000 and not 127.0.0.1:9000? ?Use > 0.0.0.0:9000 if you''re bound to that (I had another user on the > raindrops mailing list with that problem).My unicorn.rb says 127.0.0.1. But just to test I tried fetching stats for 0.0.0.0:9000 but got the same error. -Greg
ghazel at gmail.com wrote:> On Fri, Dec 3, 2010 at 1:49 PM, Eric Wong <normalperson at yhbt.net> wrote: > > ghazel at gmail.com wrote: > >> Using "modprobe inet_diag" made the script spit out a new error: > >> > >> ? ? ? ? ? ? address ? ? active ? ? queued > >> linux-tcp-listener-stats.rb:42:in `tcp_listener_stats'': NLMSG_ERROR > >> (RuntimeError) > >> ? ? ? ? from linux-tcp-listener-stats.rb:42 > > > > I don''t think I''ve ever seen NLMSG_ERROR before. ?Are you running > > this as the same user that originally bound the listener (the user > > of the Unicorn master process)? > > Yes. I also tried as root, same error. > > >> $ sudo modprobe -l inet_diag > >> /lib/modules/2.6.21.7-2.fc8xen/kernel/net/ipv4/inet_diag.ko > >> > >> I dug in to raindrops a little bit, and it seems the error associated > >> with the NLMSG_ERROR is -2, > >> "No such file or directory". > > > > Are you bound to 0.0.0.0:9000 and not 127.0.0.1:9000? ?Use > > 0.0.0.0:9000 if you''re bound to that (I had another user on the > > raindrops mailing list with that problem). > > My unicorn.rb says 127.0.0.1. But just to test I tried fetching stats > for 0.0.0.0:9000 but got the same error.Ah, can you try "modprobe tcp_diag", too? I can''t remember what other quirks there were for ancient kernels. Hopefully that works, because I''d be at a loss otherwise :x -- Eric Wong
On Sat, Dec 4, 2010 at 3:48 PM, Eric Wong <normalperson at yhbt.net> wrote:> ghazel at gmail.com wrote: >> On Fri, Dec 3, 2010 at 1:49 PM, Eric Wong <normalperson at yhbt.net> wrote: >> > ghazel at gmail.com wrote: >> >> Using "modprobe inet_diag" made the script spit out a new error: >> >> >> >> ? ? ? ? ? ? address ? ? active ? ? queued >> >> linux-tcp-listener-stats.rb:42:in `tcp_listener_stats'': NLMSG_ERROR >> >> (RuntimeError) >> >> ? ? ? ? from linux-tcp-listener-stats.rb:42 >> > >> > I don''t think I''ve ever seen NLMSG_ERROR before. ?Are you running >> > this as the same user that originally bound the listener (the user >> > of the Unicorn master process)? >> >> Yes. I also tried as root, same error. >> >> >> $ sudo modprobe -l inet_diag >> >> /lib/modules/2.6.21.7-2.fc8xen/kernel/net/ipv4/inet_diag.ko >> >> >> >> I dug in to raindrops a little bit, and it seems the error associated >> >> with the NLMSG_ERROR is -2, >> >> "No such file or directory". >> > >> > Are you bound to 0.0.0.0:9000 and not 127.0.0.1:9000? ?Use >> > 0.0.0.0:9000 if you''re bound to that (I had another user on the >> > raindrops mailing list with that problem). >> >> My unicorn.rb says 127.0.0.1. But just to test I tried fetching stats >> for 0.0.0.0:9000 but got the same error. > > Ah, can you try "modprobe tcp_diag", too? ?I can''t remember what other > quirks there were for ancient kernels. ?Hopefully that works, because > I''d be at a loss otherwise :xAh! That worked. Thanks! -Greg