asterisk users - Oct 2012 - Too many open files: what might cause this?

So a few people just reported that they couldn't make any calls.  I 
logged into asterisk and at first everything on the console looked 
normal, then I got swamped with messages about too many open files.

This is from my asterisk/messages log file:
[Oct  2 16:46:00] WARNING[19429] rtp.c: Unable to allocate RTCP socket: 
Too many open files
[Oct  2 16:46:00] WARNING[19429] udptl.c: Unable to allocate socket: Too 
many open files
[Oct  2 16:46:00] WARNING[19429] acl.c: Cannot create socket
[Oct  2 16:46:00] WARNING[19429] channel.c: Channel allocation failed: 
Can't create alert pipe! Try increasing max file descriptors with ulimit -n

Messages like that repeat a few dozen times, and then I get this one....

manager.c: Accept returned -1: Too many open files

...and that repeated tens of thousands of times.  I killed asterisk and 
restarted it.  Looks normal again.

What the heck just happened?  A bug? Was I attacked? Maybe I'm honestly 
hitting some system limit and I should bump up max file descriptors like 
the message says?  We do have a few hundred SIP peers and maybe we'll 
hit 20-30 simultaneous calls at peak times but I didn't think that was 
particularly high load.

This is Asterisk 1.4.44.  I know the 1.4 branch is old, but it had been 
trouble free for years (until now), and I'd have to rewrite some config 
syntax to upgrade so I didn't see a need to do it.

I was looking at open files.  lsof | wc -l tells me around 2000 or so.  
The total number goes up and down, but hovers around 2000 and doesn't 
seem to show any upward trend.  I haven't rebooted the system since I 
killed and restarted asterisk, so the first guess would be that asterisk 
is what had all the files open.  I wish I had checked that before I 
killed it.

cat /proc/sys/fs/file-max says 367467 so I guess whatever happened must 
have been pretty extreme.
> So a few people just reported that they couldn't make any calls.  I 
> logged into asterisk and at first everything on the console looked 
> normal, then I got swamped with messages about too many open files.
>
> This is from my asterisk/messages log file:
> [Oct  2 16:46:00] WARNING[19429] rtp.c: Unable to allocate RTCP 
> socket: Too many open files
> [Oct  2 16:46:00] WARNING[19429] udptl.c: Unable to allocate socket: 
> Too many open files
> [Oct  2 16:46:00] WARNING[19429] acl.c: Cannot create socket
> [Oct  2 16:46:00] WARNING[19429] channel.c: Channel allocation failed: 
> Can't create alert pipe! Try increasing max file descriptors with 
> ulimit -n
>
> Messages like that repeat a few dozen times, and then I get this one....
>
> manager.c: Accept returned -1: Too many open files
>
> ...and that repeated tens of thousands of times.  I killed asterisk 
> and restarted it.  Looks normal again.
>
> What the heck just happened?  A bug? Was I attacked? Maybe I'm 
> honestly hitting some system limit and I should bump up max file 
> descriptors like the message says?  We do have a few hundred SIP peers 
> and maybe we'll hit 20-30 simultaneous calls at peak times but I 
> didn't think that was particularly high load.
>
> This is Asterisk 1.4.44.  I know the 1.4 branch is old, but it had 
> been trouble free for years (until now), and I'd have to rewrite some 
> config syntax to upgrade so I didn't see a need to do it.
>
>
>
> -- 
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
> New to Asterisk? Join us for a live introductory webinar every Thurs:
>               http://www.asterisk.org/hello
>
> asterisk-users mailing list
> To UNSUBSCRIBE or update options visit:
>   http://lists.digium.com/mailman/listinfo/asterisk-users
>

Adam Moffett ????? 03.10.2012 00:09: 
> manager.c: Accept
returned -1: Too many open files

It's not a problem of old Asterisk.
You should check your system limits with the "ulimit -a" 

You can
increase different limits, say "ulimit -n 4096" will increase limit
for
open files to 4096 

I suggest reading ulimit manuals. 

-- 
With Best
Regards
Mikhail Lischuk

 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.digium.com/pipermail/asterisk-users/attachments/20121003/a3847970/attachment.htm>