Stefan Viljoen
2015-Aug-12 07:43 UTC
[asterisk-users] 786 000 files limit Centos 7 - Asterisk
Hi Markus Thanks for the reply, I have set those files previously as well... It seems the problem for me is on my Centos 7 box that the Asterisk binary does not "know" that these are the limits, and imposes a 1024 open file limit count for some reason. It seems using prlimit to force the running binary to use a higher file limit is working - but it has to be done manually each time the asterisk process is started, the prlimit cronjob I set up to force the file limit higher does not seem to work - you have to physically type in /usr/bin/prlimit --pid `pidof asterisk` --nofile=786000:786000 and running that from a script (As root) doesn't seem to work. Anyway, thanks for the advice and taking the time to reply. Kind regards, Message: 11 Date: Tue, 11 Aug 2015 18:29:01 +0200 From: Markus Weiler <markus_weiler at mailworks.org> To: asterisk-users at lists.digium.com Subject: Re: [asterisk-users] 786 000 files limit Centos 7 - Asterisk keep complaining Message-ID: <55CA22CD.4060400 at mailworks.org> Content-Type: text/plain; charset=windows-1252; format=flowed Hi Stefan, we ran into a similar problem using Debian. There we are able to check the current limits using: pidof asterisk -> 23351 cat /proc/23351/limits Output: Limit Soft Limit Hard Limit Units Max open files 1024 1024 files I think that in the end /etc/security/limits.conf * hard nofile 500000 * soft nofile 500000 root hard nofile 500000 root soft nofile 500000 did the trick. We also tried vi /etc/sysctl.conf fs.file-max = 500000 not sure what the solution in the end was. But I remember rebooting was important. Markus Am 11.08.2015 um 11:00 schrieb Stefan Viljoen:>>> Anybody else ran into this? >> No, but I would ask myself why so many file descriptors are being used. >> It sounds like you have a file descriptor leak (not being closed when >> finished with). > Hi Tony > > Thanks for replying. > > I suspected something like that, though repeatedly running > > lsof | wc -l > > Always stays quite low - 100 000 open files, which is still 8 times less > than the system maximum as confirmed by running ulimit -n > > I also note that this number will increase to about 125 000 but never go > higher than that, then, as calls hang up, decreate again - during timeswhen> the CLI is spammed with 100s of "broken pipe" errors due to insuffiecient > file descriptors, this number never reaches beyond 125 000 out of the > available 800 000 open files. > >> You might also want to look at the output of lsof (or at least some ofit)>> to see what all these file descriptors are pointing to, and whether it is >> indeed Asterisk that is consuming them. > If I grep by asterisk on the output of lsof the few thousand lines I have > looked at all seem to indicate legitimate uses - there are at least two > files for each conversation in progress (I assume for inward and outward > RTP) plus one for each file being mixmonitored (which also seems logical) > and also number-of-active-calls connections to res_timing_dahdi - whichall> looks correct... > >> If it is Asterisk, it's quite possible, even probable, that such a leak >> has been found and fixed, even in the 1.8 series. 1.8.11.0 is rather old->> the latest is 1.8.32.3, so it would be best to update to that version and >> see if the problem persists. > Ok, I will have to consider that. The thing is the problem is notconsistent> - I can (for example) run 60 calls, with no problems and no reported > failures in opening files, then calls will -decrease- to about 40 and then > later spike to 70, but around 50 calls I get the errors coming upthousands> of times in the CLI, then suddenly stop as the calls -increase- which > doesn't make sense. But this kind of behaviour does seem consistent with a > possible leak. > > SOMETHING NEW > > I have now ran > > /usr/bin/prlimit --pid `pidof asterisk` > > and I have noticed that even though I have 800 000 files specified, the > ACTUAL limit in place on Asterisk for numbers of files is only 1024?! > > # prlimit --pid `pidof asterisk` > RESOURCE DESCRIPTION SOFT HARD UNITS > AS address space limit unlimited unlimited bytes > CORE max core file size unlimited unlimited blocks > CPU CPU time unlimited unlimited seconds > DATA max data size unlimited unlimited bytes > FSIZE max file size unlimited unlimited blocks > LOCKS max number of file locks held unlimited unlimited > MEMLOCK max locked-in-memory address space 65536 65536 bytes > MSGQUEUE max bytes in POSIX mqueues 819200 819200 bytes > NICE max nice prio allowed to raise 0 0 > NOFILE max number of open files 1024 4096 > NPROC max number of processes 30861 30861 > RSS max resident set size unlimited unlimited pages > RTPRIO max real-time priority 0 0 > RTTIME timeout for real-time tasks unlimited unlimitedmicrosecs> SIGPENDING max number of pending signals 30861 30861 > STACK max stack size 8388608 unlimited bytes > > Accordingly I have put this into a cronjob ran each minute: > > prlimit --pid `pidof asterisk` --nofile=786000:786000 > > to try and force the running binary to keep a high file limit (sources say > to keep it less than the ACTUAL system file limit, in my case 800 000files)> on the live Asterisk process. > > I'll see if this maybe helps - the above runs via cron each minute. > > So it appears for some reason somehow the live running asterisk process > "loses track" of how many open files it may have, or when it starts it > somehow does not start with the correct number of maximum open files, asset> in the system / kernel config? > > Anyway, thank you for replying, I'll monitor this new "Cronjob fixup" I'm > trying and see if it helps. > > No wonder it is complaining about running out of file handles if itACTUALLY> was only using 1024! > > Kind regards > > Stefan > >------------------------------ Message: 12 Date: Tue, 11 Aug 2015 16:50:29 +0000 (UTC) From: tony at softins.co.uk (Tony Mountifield) To: asterisk-users at lists.digium.com Subject: Re: [asterisk-users] 786 000 files limit Centos 7 - Asterisk keep complaining Message-ID: <assp.06651285b6.mqd94l$qb1$1 at softins.softins.co.uk> In article <002b01d0d414$36af31b0$a40d9510$@verishare.co.za>, Stefan Viljoen <viljoens at verishare.co.za> wrote:> >> Anybody else ran into this? > > >No, but I would ask myself why so many file descriptors are being used. > >It sounds like you have a file descriptor leak (not being closed when > >finished with). > > Hi Tony > > Thanks for replying. > > I suspected something like that, though repeatedly running > > lsof | wc -l > > Always stays quite low - 100 000 open files, which is still 8 times less > than the system maximum as confirmed by running ulimit -n>From what you said below, the above is probably not relevant...> > SOMETHING NEW > > I have now ran > > /usr/bin/prlimit --pid `pidof asterisk` > > and I have noticed that even though I have 800 000 files specified, the > ACTUAL limit in place on Asterisk for numbers of files is only 1024?!Yes, this is likely. Have a look in /usr/sbin/safe_asterisk, at the commented-out settings for SYSMAXFILES and MAXFILES, and try setting those. Cheers Tony -- Tony Mountifield Work: tony at softins.co.uk - http://www.softins.co.uk Play: tony at mountifield.org - http://tony.mountifield.org ------------------------------ _______________________________________________ --Bandwidth and Colocation Provided by http://www.api-digital.com-- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users End of asterisk-users Digest, Vol 133, Issue 13 ***********************************************
Joshua Colp
2015-Aug-12 11:34 UTC
[asterisk-users] 786 000 files limit Centos 7 - Asterisk
On Wed, Aug 12, 2015, at 04:43 AM, Stefan Viljoen wrote:> Hi Markus > > Thanks for the reply, I have set those files previously as well... > > It seems the problem for me is on my Centos 7 box that the Asterisk > binary > does not "know" that these are the limits, and imposes a 1024 open file > limit count for some reason.<snip> This is actually enforced by the system, not by Asterisk itself. -- Joshua Colp Digium, Inc. | Senior Software Developer 445 Jan Davis Drive NW - Huntsville, AL 35806 - US Check us out at: www.digium.com & www.asterisk.org